Dashboard Guide

Procedure

1. Enter the following URL in a web browser:

   Syntax

   https://HOST_NAME:PORT

   Replace:

   • HOST_NAME with the fully qualified domain name (FQDN) of the active manager host.

   • PORT with port 8443

     Example

     https://host01:8443

     You can also get the URL of the dashboard by running the following command in the Cephadm shell:

     Example

     [ceph: root@host01 /]# ceph mgr services

     This command will show you all endpoints that are currently configured. Look for the dashboard key to obtain the URL for accessing the dashboard.

2. On the login page, enter the username admin and the default password provided during bootstrapping.
3. You have to change the password the first time you log in to the Red Hat Ceph Storage dashboard.

4. After logging in, the dashboard default landing page is displayed, which provides details, a high-level overview of status, performance, inventory, and capacity metrics of the Red Hat Ceph Storage cluster.

   Figure 2.1. Ceph dashboard landing page

   
5. Click the menu icon ( ) on the dashboard landing page to collapse or display the options in the vertical menu.

Procedure

1. Copy the admin key from the bootstrapped host to other hosts:

   Syntax

   ssh-copy-id -f -i /etc/ceph/ceph.pub root@HOST_NAME

   Example

   [ceph: root@host01 /]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@host02
   [ceph: root@host01 /]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@host03

2. Log in to the dashboard with the default credentials provided during bootstrap.
3. Change the password and log in to the dashboard with the new password .

4. On the landing page, click Expand Cluster.

   Note

   Clicking Expand Cluster opens a wizard taking you through the expansion steps. To skip and add hosts and services separately, click Skip.

   Figure 2.2. Expand cluster

   

5. Add hosts. This needs to be done for each host in the storage cluster.

   1. In the Add Hosts step, click Add.

   2. Provide the hostname. This is same as the hostname that was provided while copying the key from the bootstrapped host.

      Note

      Add multiple hosts by using a comma-separated list of host names, a range expression, or a comma separated range expression.

   3. Optional: Provide the respective IP address of the host.
   4. Optional: Select the labels for the hosts on which the services are going to be created. Click the pencil icon to select or add new labels.

   5. Click Add Host.

      The new host is displayed in the Add Hosts pane.

   6. Click Next.

6. Create OSDs:

   1. In the Create OSDs step, for Primary devices, Click Add.
   2. In the Primary Devices window, filter for the device and select the device.
   3. Click Add.
   4. Optional: In the Create OSDs window, if you have any shared devices such as WAL or DB devices, then add the devices.
   5. Optional: In the Features section, select Encryption to encrypt the features.
   6. Click Next.

7. Create services:

   1. In the Create Services step, click Create.

   2. In the Create Service form:

      1. Select a service type.
      2. Provide the service ID. The ID is a unique name for the service. This ID is used in the service name, which is service_type.service_id.

1. Select if the placement is by hosts or label.
2. Select the hosts.

3. In the Count field, provide the number of daemons or services that need to be deployed.

   1. Click Create Service.

      The new service is displayed in the Create Services pane.

      1. In the Create Service window, Click Next.

      2. Review the cluster expansion details.

         Review the Cluster Resources, Hosts by Services, Host Details. To edit any parameters, click Back and follow the previous steps.

         Figure 2.3. Review cluster

         

      3. Click Expand Cluster.

         The Cluster expansion displayed notification is displayed and the cluster status changes to HEALTH_OK on the dashboard.

Verification

1. Log in to the cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Run the ceph -s command.

   Example

   [ceph: root@host01 /]# ceph -s

   The health of the cluster is HEALTH_OK.

Procedure

1. View if cluster upgrades are available and upgrade as needed from Administration > Upgrade on the dashboard.

   Note

   If the dashboard displays the Not retrieving upgrades message, check if the registries were added to the container configuration files with the appropriate log in credentials to Podman or docker.

   Click Pause or Stop during the upgrade process, if needed. The upgrade progress is shown in the progress bar along with information messages during the upgrade.

   Note

   When stopping the upgrade, the upgrade is first paused and then prompts you to stop the upgrade.

2. Optional. View cluster logs during the upgrade process from the Cluster logs section of the Upgrade page.
3. Verify that the upgrade is completed successfully by confirming that the cluster status displays OK state.

Procedure

1. Log in to the dashboard:

   Syntax

   https://HOST_NAME:8443

2. Go to User→Change password on the menu bar.
3. Enter the old password, for verification.
4. In the New password field enter a new password. Passwords must contain a minimum of 8 characters and cannot be the same as the last one.
5. In the Confirm password field, enter the new password again to confirm.

6. Click Change Password.

   You will be logged out and redirected to the login screen. A notification appears confirming the password is changed.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Create the dashboard_password.yml file:

   Example

   [ceph: root@host01 /]# touch dashboard_password.yml

3. Edit the file and add the new dashboard password:

   Example

   [ceph: root@host01 /]# vi dashboard_password.yml

4. Reset the dashboard password:

   Syntax

   ceph dashboard ac-user-set-password DASHBOARD_USERNAME -i PASSWORD_FILE

   Example

   [ceph: root@host01 /]# ceph dashboard ac-user-set-password admin -i dashboard_password.yml
   {"username": "admin", "password": "$2b$12$i5RmvN1PolR61Fay0mPgt.GDpcga1QpYsaHUbJfoqaHd1rfFFx7XS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": , "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": false}

Procedure

1. As a root user, create a grafana.yml file and provide the following details:

   Syntax

   service_type: grafana
   spec:
     initial_admin_password: PASSWORD

   Example

   service_type: grafana
   spec:
     initial_admin_password: mypassword

2. Mount the grafana.yml file under a directory in the container:

   Example

   [root@host01 ~]# cephadm shell --mount grafana.yml:/var/lib/ceph/grafana.yml

   Note

   Every time you exit the shell, you have to mount the file in the container before deploying the daemon.

3. Optional: Check if the dashboard Ceph Manager module is enabled:

   Example

   [ceph: root@host01 /]# ceph mgr module ls

4. Optional: Enable the dashboard Ceph Manager module:

   Example

   [ceph: root@host01 /]# ceph mgr module enable dashboard

5. Apply the specification using the orch command:

   Syntax

   ceph orch apply -i FILE_NAME.yml

   Example

   [ceph: root@host01 /]# ceph orch apply -i /var/lib/ceph/grafana.yml

6. Redeploy grafana service:

   Example

   [ceph: root@host01 /]# ceph orch redeploy grafana

   This creates an admin user called admin with the given password and the user can log in to the Grafana URL with these credentials.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Check the Ceph Manager services:

   Example

   [ceph: root@host01 /]# ceph mgr services
   
   {
       "prometheus": "http://10.8.0.101:9283/"
   }

   You can see that the Dashboard URL is not configured.

3. Enable the dashboard module:

   Example

   [ceph: root@host01 /]# ceph mgr module enable dashboard

4. Create the self-signed certificate for the dashboard access:

   Example

   [ceph: root@host01 /]# ceph dashboard create-self-signed-cert

   Note

   You can disable the certificate verification to avoid certification errors.

5. Check the Ceph Manager services:

   Example

   [ceph: root@host01 /]# ceph mgr services
   
   {
       "dashboard": "https://10.8.0.101:8443/",
       "prometheus": "http://10.8.0.101:9283/"
   }

6. Create the admin user and password to access the Red Hat Ceph Storage dashboard:

   Syntax

   echo -n "PASSWORD" > PASSWORD_FILE
   ceph dashboard ac-user-create admin -i PASSWORD_FILE administrator

   Example

   [ceph: root@host01 /]# echo -n "p@ssw0rd" > password.txt
   [ceph: root@host01 /]# ceph dashboard ac-user-create admin -i password.txt administrator

7. Enable the monitoring stack. See the Enabling monitoring stack section in the Red Hat Ceph Storage Dashboard Guide for details.

Procedure

1. Log in to the Dashboard.

2. Click the Dashboard Settings icon and then click User management.

   
3. On Roles tab, click Create.

4. In the Create Role window, set the Name, Description, and select the Permissions for this role, and then click the Create Role button.

   

   In this example, the user assigned with ganesha-manager and rgw-manager roles can manage all NFS-Ganesha gateway and Ceph object gateway operations.

5. You get a notification that the role was created successfully.
6. Click on the Expand/Collapse icon of the row to view the details and permissions given to the roles.

Procedure

1. Log in to the Dashboard.

2. Click the Dashboard Settings icon and then click User management.

   
3. On Roles tab, click the role you want to edit.

4. In the Edit Role window, edit the parameters, and then click Edit Role.

   
5. You get a notification that the role was updated successfully.

Procedure

1. Log in to the Dashboard.

2. Click the Dashboard Settings icon and then click User management.

   
3. On Roles tab, click the role you want to clone.
4. Select Clone from the Edit drop-down menu.

5. In the Clone Role dialog box, enter the details for the role, and then click Clone Role.

   
6. Once you clone the role, you can customize the permissions as per the requirements.

Procedure

1. Log in to the Dashboard.

2. Click the Dashboard Settings icon and then select User management.

   
3. On the Roles tab, click the role you want to delete and select Delete from the action drop-down.
4. In the Delete Role notification, select Yes, I am sure and click Delete Role.

Procedure

1. Log in to the Dashboard.

2. Click the Dashboard Settings icon and then click User management.

   
3. On Users tab, click Create.

4. In the Create User window, set the Username and other parameters including the roles, and then click Create User.

   
5. You get a notification that the user was created successfully.

Procedure

1. Log in to the Dashboard.

2. Click the Dashboard Settings icon and then click User management.

   
3. To edit the user, click the row.
4. On Users tab, select Edit from the Edit drop-down menu.

5. In the Edit User window, edit parameters like password and roles, and then click Edit User.

   
   Note

   If you want to disable any user’s access to the Ceph dashboard, you can uncheck Enabled option in the Edit User window.

6. You get a notification that the user was created successfully.

Procedure

1. Log in to the Dashboard.

2. Click the Dashboard Settings icon and then click User management.

   
3. On Users tab, click the user you want to delete.
4. select Delete from the Edit drop-down menu.

5. In the Delete User notification, select Yes, I am sure and click Delete User.

   

Procedure

1. From the dashboard navigation, go to Administration→Ceph Users.
2. Click Create.

3. In the Create User form, provide the following details:

   1. User entity: Enter as TYPE.ID.
   2. Entity: This can be mon, mgr, osd, or mds.

   3. Entity Capabilities: Enter the capabilities that you can to provide to the user. For example, 'allow *' and profile crash are some of the capabilities that can be assigned to the client.

      Note

      You can add more entities to the user, based on the requirement.

      

4. Click Create User.

   A notification displays that the user is created successfully.

Procedure

1. From the dashboard navigation, go to Administration→Ceph Users.
2. Select the user whose roles you want to edit.
3. Click Edit.

4. In the Edit User form, edit the Entity and Entity Capabilities, as needed.

   Note

   You can add more entities to the user based on the requirement.

5. Click Edit User.

   A notification displays that the user is successfully edited.

Procedure

1. Create a keyring file on the local host:

   Example

   [localhost:~]$ cat import.keyring
   
   [client.test11]
   	key = AQD9S29kmjgJFxAAkvhFar6Af3AWKDY2DsULRg==
   	caps mds = "allow *"
   	caps mgr = "allow *"
   	caps mon = "allow *"
   	caps osd = "allow r"

2. From the dashboard navigation, go to Administration→Ceph Users.
3. Select the user whose roles you want to export.
4. Select Edit→Import.
5. In the Import User form, click Choose File.
6. Browse to the file on your local host and select.

7. Click Import User.

   

   A notification displays that the keys are successfully imported.

Procedure

1. From the dashboard navigation, go to Administration→Ceph Users.
2. Select the user whose roles you want to export.
3. Select Export from the action drop-down.

4. From the Ceph user export data dialog, click Copy to Clipboard.

   

   A notification displays that the keys are successfully copied.

5. On your local system, create a keyring file and paste the keys:

   Example

   [localhost:~]$ cat exported.keyring
   
   [client.test11]
   	key = AQD9S29kmjgJFxAAkvhFar6Af3AWKDY2DsULRg==
   	caps mds = "allow *"
   	caps mgr = "allow *"
   	caps mon = "allow *"
   	caps osd = "allow r"

6. Click Close.

Procedure

1. From the dashboard navigation, go to Administration→Ceph Users.
2. Select the user that you want to delete and select Delete from the action drop-down.
3. In the Delete user dialog, select Yes, I am sure..

4. Click Delete user.

   A notification displays that the user is deleted successfully.

1. From the dashboard navigation, go to Administration→Services.

2. Expand the service with the daemon that will have the action run on.

   Note

   The row can be collapsed at any time.

3. On the Daemons tab, select the row with the daemon.

   Note

   The Daemons table can be searched and filtered.

4. Select the action that needs to be run on the daemon. The options are Start, Stop, Restart, and Redeploy.

   Figure 5.1. Managing daemons from Services

   

1. From the dashboard navigation, go to Cluster→Hosts.
2. On the Hosts List tab, expand the host row and select the host with the daemon to perform the action on.

3. On the Daemons tab of the host, select the row with the daemon.

   Note

   The Daemons table can be searched and filtered.

4. Select the action that needs to be run on the daemon. The options are Start, Stop, Restart, and Redeploy.

   Figure 5.2. Managing daemons from Hosts

   

Procedure

1. From the dashboard navigation, go to Cluster→Hosts.
2. On the Hosts List tab, expand the host row and select the host with the daemon to perform the action on.

3. On the Daemons tab of the host, select the row with the daemon.

   Note

   The Daemons table can be searched and filtered.

4. Select the action that needs to be run on the daemon. The options are Start, Stop, Restart, and Redeploy.

   Figure 6.1. Monitoring hosts of the Ceph cluster

   

Procedure

1. From the dashboard navigation, go to Administration→Configuration.

2. To view the details of the configuration, expand the row contents.

   Figure 6.2. Configuration options

   
3. Optional: Use the search field to find a configuration.

4. Optional: You can filter for a specific configuration. Use the following filters:

   • Level - Basic, advanced, or dev
   • Service - Any, mon, mgr, osd, mds, common, mds_client, rgw, and similar filters.
   • Source - Any, mon, and similar filters
   • Modified - yes or no

5. To edit a configuration, select the configuration row and click Edit.

   1. Use the Edit form to edit the required parameters, and click Update.

      A notification displays that the configuration was updated successfully.

Viewing the manager modules

1. From the dashboard navigation, go to Administration→Manager Modules.

2. To view the details of a specific manager module, expand the row contents.

   Figure 6.3. Manager modules

   

Editing a manager module

1. Select the row:

   Note

   Not all modules have configurable parameters. If a module is not configurable, the Edit button is disabled.

2. Edit the required parameters and click Update.

   A notification displays that the module was updated successfully.

Procedure

1. From the dashboard navigation, go to Cluster→Monitors.

   The Monitors panel displays information about the overall monitor status and monitor hosts that are in and out of quorum.

2. To see the number of open sessions, in the In Quorum table, hover the cursor over the Open Sessions.

   

3. To see performance counters for any monitor, click the Name in the In Quorum and Not In Quorum tables.

   Figure 6.4. Viewing monitor Performance Counters

   

Procedure

1. From the dashboard navigation, go to Administration→Services.

2. Expand the service for more details.

   Figure 6.5. Monitoring services of the Ceph cluster

   

Procedure

1. From the dashboard navigation, go to Cluster→OSDs.

2. To view the details of a specific OSD, from the OSDs List tab, expand an OSD row.

   Figure 6.6. Monitoring OSDs of the Ceph cluster

   

   You can view additional details such as Devices, Attributes (OSD map), Metadata, Device Health, Performance counter, and Performance Details, by clicking on the respective tabs.

Procedure

1. Log in to the Grafana URL and select the RGW_Overview panel:

   Syntax

   https://DASHBOARD_URL:3000

   Example

   https://dashboard_url:3000

2. Verify the HAProxy metrics on the Grafana URL.
3. From the Ceph dashboard navigation, go to Object→Gateways.

4. From the Overall Performance tab, verify the Ceph Object Gateway HAProxy metrics.

   Figure 6.7. HAProxy metrics

   

Procedure

1. From the dashboard navigation, go to Cluster→CRUSH map.

2. To view the details of the specific OSD, click it’s row.

   Figure 6.8. CRUSH Map detail view

   

Procedure

1. From the dashboard navigation, go to Observability→Logs.

2. From the Cluster Logs tab, view cluster logs.

   Figure 6.9. Cluster logs

   
   1. Use the Priority filter to filter by Debug, Info, Warning, Error, or All.
   2. Use the Keyword field to enter text to search by keyword.
   3. Use the Date picker to filter by a specific date.
   4. Use the Time range fields to enter a range, using the HH:MM - HH:MM format. Hours must be entered using numbers 0 to 23.
   5. To combine filters, set two or more filters.
3. To save the logs, use the Download or Copy to Clipboard buttons.

Procedure

1. From the dashboard navigation, go to Administration→Services.
2. From Services, click Create.
3. In the Create Service form, from the Type list, select loki. Fill in the remaining details, and click Create Service.

4. Repeat the previous step to create the Promtail service. Select promtail from the Type list.

   The loki and promtail services are displayed in the Services table, after being created successfully.

   Figure 6.10. Creating Loki and Promtail services

   
   Note

   By default, Promtail service is deployed on all the running hosts.

5. Enable logging to files.

   1. Go to Administration→Configuration.
   2. Select log_to_file and click Edit.

   3. In the Edit log_to_file form, set the global value to true.

      Figure 6.11. Configuring log files

      

   4. Click Update.

      The Updated config option log_to_file notification displays and you are returned to the Configuration table.

   5. Repeat these steps for mon_cluster_log_to_file, setting the global value to true.

      Note

      Both log_to_file and mon_cluster_log_to_file files need to be configured.

6. Optional: To view the Ceph Object Gateway 'ops_log', rgw_enable_ops_log must be set to true by using the following command:

   $ ceph config set client.rgw rgw_enable_ops_log true

   To do it from the dashboard, follow the below steps:

   1. Go to Administration → Configuration.
   2. Change level from 'basic' to 'Dev'.
   3. Search for rgw_enable_ops_log and edit the value to true.
   4. Next, under the Daemon Logs tab, locate the logs file in the filename field and run the query to view the ops log.

7. To view the centralized logs, go to Observability→Logs and switch to the Daemon Logs tab. Use Log browser to select files and click Show logs to view the logs from that file.

   Figure 6.12. View centralized logs

   

Procedure

1. From the dashboard navigation, go to Cluster→Pools.
2. View the Pools List tab, which gives the details of Data protection and the application for which the pool is enabled. Hover the mouse over Usage, Read bytes, and Write bytes for the required details.

3. Expand the pool row for detailed information about a specific pool.

   Figure 6.13. Monitoring pools

   
4. For general information, go to the Overall Performance tab.

Procedure

1. From the dashboard navigation, go to File→File Systems.
2. To view more information about an individual file system, expand the file system row.

Procedure

1. From the dashboard navigation, go to Object→Gateways.
2. View information about individual gateways, from the Gateways List tab.
3. To view more information about an individual gateway, expand the gateway row.
4. If you have configured multiple Ceph Object Gateway daemons, click on Sync Performance tab and view the multi-site performance counters.

Procedure

1. From the dashboard navigation, go to Block→Images.

2. Expand the image row to see detailed information.

   Figure 6.14. Monitoring Block Device images

   

Procedure

1. Log into the cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Set the APIs for the monitoring stack:

   1. Specify the host and port of the Alertmanager server:

      Syntax

      ceph dashboard set-alertmanager-api-host ALERTMANAGER_API_HOST:PORT

      Example

      [ceph: root@host01 /]# ceph dashboard set-alertmanager-api-host http://10.0.0.101:9093
      Option ALERTMANAGER_API_HOST updated

   2. To see the configured alerts, configure the URL to the Prometheus API. Using this API, the Ceph Dashboard UI verifies that a new silence matches a corresponding alert.

      Syntax

      ceph dashboard set-prometheus-api-host PROMETHEUS_API_HOST:PORT

      Example

      [ceph: root@host01 /]# ceph dashboard set-prometheus-api-host http://10.0.0.101:9095
      Option PROMETHEUS_API_HOST updated

      After setting up the hosts, refresh your browser’s dashboard window.

      • Specify the host and port of the Grafana server:

        Syntax

        ceph dashboard set-grafana-api-url GRAFANA_API_URL:PORT

        Example

        [ceph: root@host01 /]# ceph dashboard set-grafana-api-url https://10.0.0.101:3000
        Option GRAFANA_API_URL updated

3. Get the Prometheus, Alertmanager, and Grafana API host details:

   Example

   [ceph: root@host01 /]# ceph dashboard get-alertmanager-api-host
   http://10.0.0.101:9093
   [ceph: root@host01 /]# ceph dashboard get-prometheus-api-host
   http://10.0.0.101:9095
   [ceph: root@host01 /]# ceph dashboard get-grafana-api-url
   http://10.0.0.101:3000

4. Optional: If you are using a self-signed certificate in your Prometheus, Alertmanager, or Grafana setup, disable the certificate verification in the dashboard This avoids refused connections caused by certificates signed by an unknown Certificate Authority (CA) or that do not match the hostname.

   • For Prometheus:

     Example

     [ceph: root@host01 /]# ceph dashboard set-prometheus-api-ssl-verify False

   • For Alertmanager:

     Example

     [ceph: root@host01 /]# ceph dashboard set-alertmanager-api-ssl-verify False

   • For Grafana:

     Example

     [ceph: root@host01 /]# ceph dashboard set-grafana-api-ssl-verify False

5. Get the details of the self-signed certificate verification setting for Prometheus, Alertmanager, and Grafana:

   Example

   [ceph: root@host01 /]# ceph dashboard get-prometheus-api-ssl-verify
   [ceph: root@host01 /]# ceph dashboard get-alertmanager-api-ssl-verify
   [ceph: root@host01 /]# ceph dashboard get-grafana-api-ssl-verify

6. Optional: If the dashboard does not reflect the changes, you have to disable and then enable the dashboard:

   Example

   [ceph: root@host01 /]# ceph mgr module disable dashboard
   [ceph: root@host01 /]# ceph mgr module enable dashboard

Procedure

1. Log into the cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Configure the custom certificate for Grafana:

   Example

   [ceph: root@host01 /]# ceph config-key set mgr/cephadm/grafana_key -i $PWD/key.pem
   [ceph: root@host01 /]# ceph config-key set mgr/cephadm/grafana_crt -i $PWD/certificate.pem

3. If Grafana is already deployed, then run reconfig to update the configuration:

   Example

   [ceph: root@host01 /]# ceph orch reconfig grafana

4. Every time a new certificate is added, follow the below steps:

   1. Make a new directory

      Example

      [root@host01 ~]# mkdir /root/internalca
      [root@host01 ~]# cd /root/internalca

   2. Generate the key:

      Example

      [root@host01 internalca]# openssl ecparam -genkey -name secp384r1 -out $(date +%F).key

   3. View the key:

      Example

      [root@host01 internalca]# openssl ec -text -in $(date +%F).key | less

   4. Make a request:

      Example

      [root@host01 internalca]# umask 077; openssl req -config openssl-san.cnf -new -sha256 -key $(date +%F).key -out $(date +%F).csr

   5. Review the request prior to sending it for signature:

      Example

      [root@host01 internalca]# openssl req -text -in $(date +%F).csr | less

   6. As the CA sign:

      Example

      [root@host01 internalca]# openssl ca -extensions v3_req -in $(date +%F).csr -out $(date +%F).crt -extfile openssl-san.cnf

   7. Check the signed certificate:

      Example

      [root@host01 internalca]# openssl x509 -text -in $(date +%F).crt -noout | less

Procedure

1. Log into the cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Configure the Alertmanager to use the webhook for notification:

   Syntax

   service_type: alertmanager
   spec:
     user_data:
       default_webhook_urls:
       - "_URLS_"

   The default_webhook_urls is a list of additional URLs that are added to the default receivers' webhook_configs configuration.

   Example

   service_type: alertmanager
   spec:
     user_data:
       webhook_configs:
       - url: 'http:127.0.0.10:8080'

3. Update Alertmanager configuration:

   Example

   [ceph: root@host01 /]#  ceph orch reconfig alertmanager

Procedure

1. From the dashboard navigation, go to Observability→Alerts.
2. View active Prometheus alerts from the Active Alerts tab.

3. View all alerts from the Alerts tab.

   To view alert details, expand the alert row.

4. To view the source of an alert, click on its row, and then click Source.

   

Procedure

1. From the dashboard navigation, go to Observability→Alerts.
2. On the Silences tab, click Create.

3. In the Create Silence form, fill in the required fields.

   1. Use the Add matcher to add silence requirements.

      Figure 7.2. Creating a silence

      

4. Click Create Silence.

   A notification displays that the silence was created successfully and the Alerts Silenced updates in the Silences table.

Procedure

1. From the dashboard navigation, go to Observability→Alerts.
2. On the Silences tab, select the row with the alert that you want to recreate, and click Recreate from the action drop-down.

3. Edit any needed details, and click Recreate Silence button.

   A notification displays indicating that the silence was edited successfully and the status of the silence is now active.

Procedure

1. Log in to the Dashboard.
2. On the navigation menu, click Cluster.
3. Select Monitoring from the drop-down menu.
4. Click the Silences tab.
5. To edit the silence, click it’s row.
6. In the Edit drop-down menu, select Edit.

7. In the Edit Silence window, update the details and click Edit Silence.

   Figure 7.3. Edit silence

   
8. You get a notification that the silence was updated successfully.

Procedure

1. From the dashboard navigation, go to Observability→Alerts.
2. On the Silences tab, select the row with the alert that you want to expire, and click Expire from the action drop-down.

3. In the Expire Silence notification, select Yes, I am sure and click Expire Silence.

   A notification displays indicating that the silence was expired successfully and the Status of the alert is expired, in the Silences table.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Create the RADOS pool, namespace, and enable rgw:

   Syntax

   ceph osd pool create POOL_NAME _
   ceph osd pool application enable POOL_NAME freeform/rgw/rbd/cephfs/nfs

   Example

   [ceph: root@host01 /]# ceph osd pool create nfs-ganesha
   [ceph: root@host01 /]# ceph osd pool application enable nfs-ganesha rgw

3. Deploy NFS-Ganesha gateway using placement specification in the command line interface:

   Syntax

   ceph orch apply nfs SERVICE_ID --placement="NUMBER_OF_DAEMONS HOST_NAME_1 HOST_NAME_2 HOST_NAME_3"

   Example

   [ceph: root@host01 /]# ceph orch apply nfs foo --placement="2 host01 host02"

   This deploys an NFS-Ganesha cluster nfsganesha with one daemon on host01 and host02.

4. Update ganesha-clusters-rados-pool-namespace parameter with the namespace and the service_ID:

   Syntax

   ceph dashboard set-ganesha-clusters-rados-pool-namespace POOL_NAME/SERVICE_ID

   Example

   [ceph: root@host01 /]# ceph dashboard set-ganesha-clusters-rados-pool-namespace nfs-ganesha/foo

5. From the dashboard navigation, go to Object→NFS.
6. Click Create.

7. Complete the Create NFS export form and click Create NFS export to save and continue.

   

8. Verify the NFS daemon is configured:

   Example

   [ceph: root@host01 /]# ceph -s

9. As a root user, check if the NFS-service is active and running:

   Example

   [root@host01 ~]# systemctl list-units | grep nfs

10. Mount the NFS export and perform a few I/O operations.
11. Once the NFS service is up and running, in the NFS-RGW container, comment out the dir_chunk=0 parameter in etc/ganesha/ganesha.conf file. Restart the NFS-Ganesha service. This allows proper listing at the NFS mount.

Procedure

1. Log in to the cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Create the CephFS storage in the backend:

   Syntax

   ceph fs volume create CEPH_FILE_SYSTEM

   Example

   [ceph: root@host01 /]# ceph fs volume create cephfs

3. Enable the Ceph Manager NFS module:

   Example

   [ceph: root@host01 /]# ceph mgr module enable nfs

4. Create an NFS Ganesha cluster:

   Syntax

   ceph nfs cluster create NFS_CLUSTER_NAME "HOST_NAME_PLACEMENT_LIST"

   Example

   [ceph: root@host01 /]# ceph nfs cluster create nfs-cephfs host02
   NFS Cluster Created Successfully

5. Get the dashboard URL:

   Example

   [ceph: root@host01 /]# ceph mgr services
   {
       "dashboard": "https://10.00.00.11:8443/",
       "prometheus": "http://10.00.00.11:9283/"
   }

6. Log in to the Ceph dashboard with your credentials.
7. On the dashboard landing page, click NFS.
8. Click Create.

9. Complete the Create NFS export form and click Create NFS export to save and continue.

   Figure 8.1. CephFS NFS export form

   

10. As a root user on the client host, create a directory and mount the NFS export:

    Syntax

    mkdir -p /mnt/nfs/
    mount -t nfs -o port=2049 HOSTNAME:EXPORT_NAME _MOUNT_DIRECTORY_

    Example

    [root@ client ~]# mkdir -p /mnt/nfs/
    [root@ client ~]# mount -t nfs -o port=2049 host02:/export1 /mnt/nfs/

Procedure

1. From the dashboard navigation, go to Object → NFS.
2. Select the row that needs to be edited. and click Edit.
3. In the Edit NFS export window, edit the required parameters.

4. Complete by clicking Edit NFS export.

   A notification displays that the NFS object was updated successfully.

Procedure

1. From the dashboard navigation, go to File→NFS.
2. Select the row that needs to be deleted and click Delete from the action drop-down.
3. In the Delete NFS export notification, select Yes, I am sure and click Delete NFS export.

Procedure

1. From the dashboard navigation, go to Cluster→Pools.
2. Click Create.

3. Fill out the Create Pool form.

   Figure 9.1. Creating pools

   
   Note

   The form changes based off of selection. Not all fields are mandatory.

   • Set the name of the pool and select the pool type.
   • Select the Pool type, either replicated or erasure. Erasure is referred to as Erasure Coded (EC).
   • Optional: Select if the PG Autoscale is on, off, or warn.
   • Optional: If using a replicated pool type, set the replicated size.
   • Optional: If using an EC pool type configure the following additional settings.
   • Optional: To see the settings for the currently selected EC profile, click the question mark.
   • Optional: Add a new EC profile by clicking the plus symbol.
   • Optional: Click the pencil symbol to select an application for the pool.
   • Optional: Set the CRUSH rule, if applicable.
   • Optional: If compression is required, select passive, aggressive, or force.
   • Optional: Set the Quotas.
   • Optional: Set the Quality of Service configuration.

4. To save the changes and complete creating the pool, click Create Pool.

   A notification displays that the pool was created successfully.

Procedure

1. From the dashboard navigation, go to Cluster→Pools.
2. To edit the pool, select the pool row and click Edit.
3. In the Edit Pool form, edit the required parameters.

4. Save changes, by clicking Edit Pool.

   A notification displays that the pool was updated successfully.

Procedure

1. From the dashboard navigation, go to Administration→Configuration.

2. From the Configuration table, select mon_allow_pool_delete, and click Edit.

   Note

   If needed, clear filters and search for the configuration.

3. From the Edit mon_allow_pool_delete form, in Values, set all values to true.

4. Click Update.

   A notification displays that the configuration was updated successfully.

5. Go to Cluster→Pools.
6. Select the pool to be deleted, and click Delete from the action drop-down.

7. In the Delete Pool dialog, select Yes, I am sure. and complete, by clicking Delete Pool.

   A notification displays that the pool was deleted successfully.

Procedure

1. From the dashboard navigation, go to Cluster→Hosts.

2. Select the host to enter maintenance mode, and click Enter Maintenance from the action drop-down.

   Note

   If the host contains Ceph Object Gateway (RGW) daemons, a warning displays that removing RGW daemons can cause clients to lose connectivity. Click Continue to enter maintenance.

   Note

   When a host enters maintenance mode, all daemons are stopped. Check the status of the daemons of a host by expanding the host view and switching to the Daemons tab.

   A notification displays that the host was moved to maintance successfully.

Verification

1. The maintenance label displays in the Status column of the Host table.

   Note

   If the maintenance mode fails, a notification displays, indicating the reasons for failure.

Procedure

1. From the dashboard navigation, go to Cluster→Hosts.

2. Select the host currently in maintenance mode, and click Exit Maintenance from the action drop-down.

   Note

   Identify which host is maintenance by checking for the maintenance label in the Status column of the Host table.

   A notification displays that the host was moved out of maintenance successfully.

3. Create the required services on the host. By default, crash and node-exporter get deployed.

Verification

1. The maintenance label is removed from the Status column of the Host table.

Procedure

1. From the dashboard navigation, go to Cluster→Hosts.

2. Select the host that is to be removed, and click Start Drain from the action drop-down.

   Figure 10.1. Selecting Start Drain option

   

   This option drains all the daemons from the host.

   Note

   The _no_schedule label is automatically applied to the host, which blocks the deployment of daemons on this host.

   1. Optional: To stop the draining of daemons from the host, click Stop Drain from the action drop-down.

3. Check that all the daemons are removed from the host.

   1. Expand the host row.

   2. Go to the Daemons tab.

      No daemons should be listed.

      Important

      A host can be safely removed from the cluster after all the daemons are removed from it.

4. Select the host that is to be removed, and click Remove from the action drop-down.

   1. In the Remove Host notification, select Yes, I am sure and click Remove Host.

      

      A notification displays that the host is removed successfully.

Creating an OSD

1. To create the OSD, from the OSDs List table, click Create.

   Figure 11.1. Add device for OSDs

   
   Note

   Ensure you have an available host and a few available devices. Check for available devices in Cluster→Physical Disks and filter for Available.

   1. In the Create OSDs form, in the Deployment Options section, select one of the following options:

      • Cost/Capacity-optimized: The cluster gets deployed with all available HDDs.
      • Throughput-optimized: Slower devices are used to store data and faster devices are used to store journals/WALs.
      • IOPS-optmized: All the available NVMe devices are used to deploy OSDs.

   2. In the Advanced Mode section, add primary, WAL, and DB devices by clicking Add.

      • Primary devices: Primary storage devices contain all OSD data.
      • WAL devices: Write-Ahead-Log devices are used for BlueStore’s internal journal and are used only if the WAL device is faster than the primary device. For example, NVMe or SSD devices.
      • DB devices: DB devices are used to store BlueStore’s internal metadata and are used only if the DB device is faster than the primary device. For example, NVMe or SSD devices.
   3. To encrypt your data, for security purposes, from the Features section of the form, select Encryption.
   4. Click Preview.

   5. In the OSD Creation Preview dialog review the OSD and click Create.

      A notification displays that the OSD was created successfully and the OSD status changes from in and down to in and up.

Editing an OSD

1. To edit an OSD, select the row and click Edit.

   1. From the Edit OSD form, edit the device class.

   2. Click Edit OSD.

      Figure 11.2. Edit an OSD

      

      A notification displays that the OSD was updated successfully.

Marking the OSD flags

1. To mark the flag of the OSD, select the row and click Flags from the action drop-down.
2. In the Individual OSD Flags form, select the OSD flags needed.

3. Click Update.

   Figure 11.3. Marking OSD flags

   

   A notification displays that the OSD flags updated successfully.

Scrubbing an OSD

1. To scrub an OSD, select the row and click Scrub from the action drop-down.

2. In the OSDs Scrub notification, click Update.

   Figure 11.4. Scrubbing an OSD

   

   A notification displays that the scrubbing of the OSD was initiated successfully.

Deep-scrubbing the OSDs

1. To deep-scrub the OSD, select the row and click Deep Scrub from the action drop-down.

2. In the OSDs Deep Scrub notification, click Update.

   Figure 11.5. Deep-scrubbing an OSD

   

   A notification displays that the deep scrubbing of the OSD was initiated successfully.

Reweighting the OSDs

1. To reweight the OSD, select the row and click Reweight from the action drop-down.
2. In the Reweight OSD form enter a value between 0 and 1.

3. Click Reweight.

   Figure 11.6. Reweighting an OSD

   

Marking OSDs out

1. To mark an OSD as out, select the row and click Mark Out from the action drop-down.

2. In the Mark OSD out notification, click Mark Out.

   Figure 11.7. Marking OSDs out

   

   The OSD status changes to out.

Marking OSDs in

1. To mark an OSD as in, select the OSD row that is in out status and click Mark In from the action drop-down.

2. In the Mark OSD in notification, click Mark In.

   Figure 11.8. Marking OSDs in

   

   The OSD status changes to in.

Marking OSDs down

1. To mark an OSD down, select the row and click Mark Down from the action drop-down.

2. In the Mark OSD down notification, click Mark Down.

   Figure 11.9. Marking OSDs down

   

   The OSD status changes to down.

Marking OSDs lost

1. To mark an OSD lost, select the OSD in out and down status and click Mark Lost from the action drop-down.

2. In the Mark OSD Lost notification, select Yes, I am sure and click Mark Lost.

   Figure 11.10. Marking OSDs lost

   

Purging OSDs

1. To purge an OSD, select the OSD in down status and click Purge from the action drop-down.

2. In the Purge OSDs notification, select Yes, I am sure and click Purge OSD.

   Figure 11.11. Purging OSDs

   

   All the flags are reset and the OSD is back in in and up status.

Destroying OSDs

1. To destroy an OSD, select the OSD in down status and click Destroy from the action drop-down.

2. In the Destroy OSDs notification, select Yes, I am sure and click Destroy OSD.

   Figure 11.12. Destroying OSDs

   

   The OSD status changes to destroyed.

Deleting OSDs

1. To delete an OSD, select the OSD and click Delete from the action drop-down.

2. In the Delete OSDs notification, select Yes, I am sure and click Delete OSD.

   Note

   You can preserve the OSD_ID when you have to to replace the failed OSD.

   Figure 11.13. Deleting OSDs

   

Procedure

1. On the dashboard, you can identify the failed OSDs in the following ways:

   • Dashboard AlertManager pop-up notifications.
   • Dashboard landing page showing HEALTH_WARN status.
   • Dashboard landing page showing failed OSDs.

   • Dashboard OSD list showing failed OSDs.

     In the following example, you can see that one of the OSDs is down and one is out on the landing page of the dashboard.

     Figure 11.14. OSD status on the Ceph Dashboard landing page

     

     You can also view the LED blinking lights on the physical drive if one of the OSDs is down.

2. From Cluster→OSDs, on the OSDs List table, select the out and down OSD.

   1. Click Flags from the action drop-down, select No Up in the Individual OSD Flags form, and click Update.
   2. Click Delete from the action drop-down. In the Delete OSD notification, select Preserve OSD ID(s) for replacement and Yes, I am sure and click Delete OSD.
   3. Wait until the status of the OSD changes to out and destroyed.

3. Optional: To change the No Up Flag for the entire cluster, from the Cluster-wide configuration menu, select Flags.

   1. In Cluster-wide OSDs Flags form, select No Up and click Update.

4. Optional: If the OSDs are down due to a hard disk failure, replace the physical drive:

   • If the drive is hot-swappable, replace the failed drive with a new one.
   • If the drive is not hot-swappable and the host contains multiple OSDs, you might have to shut down the whole host and replace the physical drive. Consider preventing the cluster from backfilling. See the Stopping and Starting Rebalancing chapter in the Red Hat Ceph Storage Troubleshooting Guide for details.
   • When the drive appears under the /dev/ directory, make a note of the drive path.
   • If you want to add the OSD manually, find the OSD drive and format the disk.

   • If the new disk has data, zap the disk:

     Syntax

     ceph orch device zap HOST_NAME PATH --force

     Example

     ceph orch device zap ceph-adm2 /dev/sdc --force

5. From the Ceph Dashboard OSDs List, click Create.

6. In the Create OSDs form Advanced Mode section, add a primary device.

   1. In the Primary devices dialog, select a Hostname filter.

   2. Select a device type from the list.

      Note

      You have to select the Hostname first and then at least one filter to add the devices.

      For example, from Hostname list, select Type and then hdd.

   3. Select Vendor and from device list, select ATA.

      Figure 11.15. Using the Primary devices filter

      
   4. Click Add.
   5. In the Create OSDs form, click Preview.

   6. In the OSD Creation Preview dialog, click Create.

      A notification displays that the OSD is created successfully and the OSD changes to be in the out and down status.

7. Select the newly created OSD that has out and down status.

   1. Click Mark In from the action drop-down.

   2. In the Mark OSD in notification, click Mark In.

      The OSD status changes to in.

   3. Click Flags from the action drop-down.
   4. Clear the No Up selection and click Update.

8. Optional: If you have changed the No Up flag before for cluster-wide configuration, in the Cluster-wide configuration menu, select Flags.

   1. In Cluster-wide OSDs Flags form, clear the No Up selection and click Update.

Verification

1. Verify that the OSD that was destroyed is created on the device and the OSD ID is preserved.

   

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Set up the credentials manually:

   Example

   [ceph: root@host01 /]# ceph dashboard set-rgw-credentials

   This creates a Ceph Object Gateway user with UID dashboard for each realm in the system.

3. Optional: If you have configured a custom admin resource in your Ceph Object Gateway admin API, you have to also set the the admin resource:

   Syntax

   ceph dashboard set-rgw-api-admin-resource RGW_API_ADMIN_RESOURCE

   Example

   [ceph: root@host01 /]# ceph dashboard set-rgw-api-admin-resource admin
   Option RGW_API_ADMIN_RESOURCE updated

4. Optional: If you are using HTTPS with a self-signed certificate, disable certificate verification in the dashboard to avoid refused connections.

   Refused connections can happen when the certificate is signed by an unknown Certificate Authority, or if the host name used does not match the host name in the certificate.

   Syntax

   ceph dashboard set-rgw-api-ssl-verify false

   Example

   [ceph: root@host01 /]# ceph dashboard set-rgw-api-ssl-verify False
   Option RGW_API_SSL_VERIFY updated

5. Optional: If the Object Gateway takes too long to process requests and the dashboard runs into timeouts, you can set the timeout value:

   Syntax

   ceph dashboard set-rest-requests-timeout _TIME_IN_SECONDS_

   The default value of 45 seconds.

   Example

   [ceph: root@host01 /]# ceph dashboard set-rest-requests-timeout 240

Procedure

1. From the dashboard navigation, go to Administration→Services.
2. Click Create.

3. Fill in the Create Service form.

   1. Select rgw from the Type service list.
   2. Enter the ID that is used in service_id.
   3. Select SSL.

   4. Click Choose File and upload the SSL certificate .pem format.

      Figure 12.1. Creating Ceph Object Gateway service

      
   5. Click Create Service.
4. Check the Ceph Object Gateway service is up and running.

Procedure

1. From the dashboard navigation, go to Administration→Services.
2. Click Create.
3. In the Create Service form, select ingress service.

4. Select backend service and edit the required parameters.

   Figure 12.2. Creating ingress service

   

5. Click Create Service.

   A notification displays that the ingress service was created successfully.

Procedure

1. From the dashboard navigation of the secondary site, go to Object→Buckets.

2. View the Object Gateway buckets on the secondary landing page that were created for the Object Gateway users on the primary site.

   Figure 12.12. Multi-site Object Gateway monitoring

   

Procedure

1. Log in to the Grafana URL.

   Syntax

   https://DASHBOARD_URL:3000

   Example

   https://dashboard_url:3000

2. Go to the 'Dashboard' tab and search for 'RGW S3 Analytics'.

3. To view per-bucket Ceph Object gateway operations, select the 'Bucket' panel:

   

4. To view user-level Ceph Object gateway operations, select the 'User' panel: