Dashboard Guide

Procedure

1. On the Hosts page, add a host and set the Metrics role.

   
2. Click Add.
3. Complete the remaining Cockpit Ceph Installer prompts.

4. After the deploy process finishes, click the Complete button at the bottom right corner of the page. This opens a window which displays the output of the command ceph status, as well as dashboard access information.

   

5. At the bottom of the Ceph Cluster Status window, the dashboard access information is displayed, including the URL, user name, and password. Take note of this information.

   
   • For more information, see Installing Red Hat Ceph Storage using the Cockpit Web User Interface in the Installation Guide.

Procedure

1. Ensure a [grafana-server] group with a node defined under it exists in the Ansible inventory file. Grafana and Prometheus are installed on this node.

   [root@jb-ceph4-admin ~]# grep grafana-server -A 1 /etc/ansible/hosts
   [grafana-server]
   jb-ceph4-mon

2. In the all.yml Ansible playbook, ensure dashboard_enabled: has not been set to False. There should be a comment indicating the default setting of True.

   [root@jb-ceph4-admin ~]# grep "dashboard_enabled" /usr/share/ceph-ansible/group_vars/all.yml
   #dashboard_enabled: True

3. Complete the rest of the steps necessary to install Ceph as outlined in the Installation Guide.

4. After running ansible-playbook site.yml for bare metal installs, or ansible-playbook site-docker.yml for container installs, Ansible will print the dashboard access information. Find the dashboard URL, username, and password towards the end of the playbook output:

   2019-12-13 15:31:17,871 p=11421 u=admin |  TASK [ceph-dashboard : print dashboard URL] ************************************************************
   2019-12-13 15:31:17,871 p=11421 u=admin |  task path: /usr/share/ceph-ansible/roles/ceph-dashboard/tasks/main.yml:5
   2019-12-13 15:31:17,871 p=11421 u=admin |  Friday 13 December 2019  15:31:17 -0500 (0:00:02.189)       0:04:25.380 *******
   2019-12-13 15:31:17,934 p=11421 u=admin |  ok: [jb-ceph4-mon] =>
     msg: The dashboard has been deployed! You can access your dashboard web UI at http://jb-ceph4-mon:8443/ as an 'admin' user with 'p@ssw0rd' password.

   Take note of the output You can access your dashboard web UI at http://jb-ceph4-mon:8443/ as an 'admin' user with 'p@ssw0rd' password.

Procedure

1. Get the URL for accessing the dashboard:

   Example

   [root@admin ~]# ceph mgr services

2. Get the current IP and port configuration of the ceph-mgr daemon:

   Example

   [root@admin ~]# netstat -ntlp

3. Set the IP address and the port:

   Syntax

   ceph config set mgr mgr/dashboard/server_addr IP_ADDRESS
   ceph config set mgr mgr/dashboard/server_port PORT
   ceph config set mgr mgr/dashboard/ssl_server_port PORT

   Example

   [root@mon ~]# ceph config set mgr mgr/dashboard/server_addr 192.168.0.120
   [root@mon ~]# ceph config set mgr mgr/dashboard/server_port 8443
   [root@mon ~]# ceph config set mgr mgr/dashboard/ssl_server_port 8443

4. Optional: Since the ceph-mgr hosts its own instance of the dashboard, you can configure them separately. Change the IP address and port for a specific manager instance:

   Syntax

   ceph config set mgr mgr/dashboard/NAME/server_addr IP_ADDRESS
   ceph config set mgr mgr/dashboard/NAME/server_port PORT
   ceph config set mgr mgr/dashboard/NAME/ssl_server_port PORT

   Replace: NAME with the ID of the ceph-mgr instance hosting the dashboard.

   Example

   [root@mon ~]# ceph config set mgr mgr/dashboard/mgrs-0/server_addr 192.168.0.120
   [root@mon ~]# ceph config set mgr mgr/dashboard/mgrs-0/server_port 8443
   [root@mon ~]# ceph config set mgr mgr/dashboard/mgrs-0/ssl_server_port 8443

Procedure

1. Enter the following URL in a web browser:

   http://HOST_NAME:PORT

   Replace:

   • HOST_NAME with the host name of the dashboard node.

   • PORT with port 8443

     For example:

     http://dashboard:8443

2. On the login page, enter the username admin and the default password p@ssw0rd if you did not change the password during installation.

   Figure 2.1. Ceph Dashboard Login Page

   

3. After logging in, the dashboard default landing page is displayed, which provides a high-level overview of status, performance, and capacity metrics of the Red Hat Ceph Storage cluster.

   Figure 2.2. Ceph Dashboard Default Landing Page

   

Procedure

1. Open the Ansible playbook file /usr/share/ceph-ansible/group_vars/all.yml for editing.

2. Uncomment and update the password on this line:

   #dashboard_admin_password: p@ssw0rd

   to:

   dashboard_admin_password: NEW_PASSWORD

   Replace NEW_PASSWORD with your preferred password.

3. Rerun the Ansible playbook file which deploys or updates the Ceph cluster.

   1. For bare metal installs, use the site.yml playbook:

      [admin@admin ceph-ansible]$ ansible-playbook -v site.yml

   2. For container installs, use the site-docker.yml playbook:

      [admin@admin ceph-ansible]$ ansible-playbook -v site-docker.yml

4. Log in using the new password.

Procedure

1. Update the password in the group_vars/all.yml file to prevent the password from being reset to p@ssw0rd when Ansible is used to reconfigure the Ceph cluster.

   1. Open the Ansible playbook file /usr/share/ceph-ansible/group_vars/all.yml for editing.

   2. Uncomment and update the password on this line:

      #dashboard_admin_password: p@ssw0rd

      to:

      dashboard_admin_password: NEW_PASSWORD

      Replace NEW_PASSWORD with your preferred password.

2. Change the password in the dashboard web user-interface.

   1. Log in to the dashboard:

      http://HOST_NAME:8443

   2. At the top right hand side toolbar, click the dashboard settings icon and then click User management.

      

   3. Locate the admin user in the Username table and click on admin.

      
   4. Above the table title Username, click on the Edit button.

   5. Enter the new password and confirm it by reentering it and click Edit User.

      

      You will be logged out and taken to the log in screen. A notification will appear confirming the password change.

3. Log back in using the new password.

Procedure

1. Optional: If you do not know which node the Grafana container is running on, find the node listed under [grafana-server] in the Ansible hosts file, usually located at /etc/ansible/hosts:

   Example

   [grafana-server]
   grafana

2. On the node where the Grafana container is running, change the password:

   Syntax

   podman exec CONTAINER_ID grafana-cli admin reset-admin-password --homepath "/usr/share/grafana" NEW_PASSWORD

   Change CONTAINER_ID to the ID of the Grafana container. Change NEW_PASSWORD to the desired Grafana password.

   Example

   [root@grafana ~]# podman exec 3f28b0309aee grafana-cli admin reset-admin-password --homepath "/usr/share/grafana" NewSecurePassword
   t=2020-10-29T17:45:58+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
   t=2020-10-29T17:45:58+0000 lvl=info msg="Starting DB migration" logger=migrator
   
   Admin password changed successfully ✔

3. On the Ansible administration node, use ansible-vault to encrypt the Grafana password, and then add the encrypted password to group_vars/all.yml.

   1. Change to the /usr/share/ceph-ansible/ directory:

      [admin@admin ~]$ cd /usr/share/ceph-ansible/

   2. Run ansible-vault and create a new vault password:

      Example

      [admin@admin ceph-ansible]$ ansible-vault encrypt_string --stdin-name 'grafana_admin_password_vault'
      New Vault password:

   3. Re-enter the password to confirm it:

      Example

      [admin@admin ceph-ansible]$ ansible-vault encrypt_string --stdin-name 'grafana_admin_password_vault'
      New Vault password:
      Confirm New Vault password:

   4. Enter the Grafana password, press enter, and then enter CTRL+D to complete the entry:

      Syntax

      ansible-vault encrypt_string --stdin-name 'grafana_admin_password_vault'
      New Vault password:
      Confirm New Vault password:
      Reading plaintext input from stdin. (ctrl-d to end input)
      NEW_PASSWORD

      Replace NEW_PASSWORD with the Grafana password that was set earlier.

      Example

      [admin@admin ceph-ansible]$ ansible-vault encrypt_string --stdin-name 'grafana_admin_password_vault'
      New Vault password:
      Confirm New Vault password:
      Reading plaintext input from stdin. (ctrl-d to end input)
      NewSecurePassword

   5. Take note of the output that begins with grafana_admin_password_vault: !vault | and ends with a few lines of numbers, as it will be used in the next step:

      Example

      [admin@admin ceph-ansible]$ ansible-vault encrypt_string --stdin-name 'grafana_admin_password_vault'
      New Vault password:
      Confirm New Vault password:
      Reading plaintext input from stdin. (ctrl-d to end input)
      NewSecurePassword
      grafana_admin_password_vault: !vault |
                $ANSIBLE_VAULT;1.1;AES256
                38383639646166656130326666633262643836343930373836376331326437353032376165306234
                3161386334616632653530383231316631636462363761660a373338373334663434363865356633
                66383963323033303662333765383938353630623433346565363534636434643634336430643438
                6134306662646365370a343135316633303830653565633736303466636261326361333766613462
                39353365343137323163343937636464663534383234326531666139376561663532
      Encryption successful

   6. Open for editing group_vars/all.yml and paste the output from above into the file:

      Example

      grafana_admin_password_vault: !vault |
                $ANSIBLE_VAULT;1.1;AES256
                38383639646166656130326666633262643836343930373836376331326437353032376165306234
                3161386334616632653530383231316631636462363761660a373338373334663434363865356633
                66383963323033303662333765383938353630623433346565363534636434643634336430643438
                6134306662646365370a343135316633303830653565633736303466636261326361333766613462
                39353365343137323163343937636464663534383234326531666139376561663532

   7. Add a line below the encrypted password with the following:

      Example

      grafana_admin_password: "{{ grafana_admin_password_vault }}"

      Note

      Using two variables as seen above is required due to a bug in Ansible that breaks the string type when assigning the vault value directly to the Ansible variable.

   8. Save and close the file.

4. Re-run ansible-playbook.

   1. For container based deployments:

      Example

      [admin@node1 ceph-ansible]$ ansible-playbook --ask-vault-pass -v site-container.yml -i hosts

      Note that -i hosts is only necessary if you are not using the default Ansible hosts file location of /etc/ansible/hosts.

   2. For bare-metal, RPM based deployments:

      Example

      [admin@node1 ceph-ansible]$ ansible-playbook --ask-vault-pass -v site.yml -i hosts

      Note that -i hosts is only necessary if you are not using the default Ansible hosts file location of /etc/ansible/hosts.

Procedure

1. Download the Red Hat Single Sign-On 7.4.0 Server on the system where Red Hat Ceph Storage is installed.

2. Unzip the folder:

   [root@cephuser]# unzip rhsso-7.4.0.zip

3. Navigate to the standalone/configuration directory and open the standalone.xml for editing:

   [root@cephuser]# cd standalone/configuration
   [root@cephuser configuration]# vi standalone.xml

4. Replace three instances of localhost and two instances of 127.0.0.1 with the IP address of the machine where Red Hat Single Sign-On is installed.

5. Optional: For Red Hat Enterprise Linux 8, users might get Certificate Authority (CA) issues. Import the custom certificates from CA and move them into the keystore with exact java version.

   Example

   [root@cephuser]# keytool -import -noprompt -trustcacerts -alias ca -file ../ca.cer -keystore /etc/java/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.272.b10-3.el8_3.x86_64/lib/security/cacert

6. To start the server from the bin directory of rh-sso-7.4 folder, run the standalone boot script:

   [root@cephuser bin]# ./standalone.sh

7. Create the admin account in http:_IP_ADDRESS_:8080/auth with a username and password:

   
   Note

   The admin account has to be created only the first time you log into the console.

8. Log into the admin console with the credentials created:

   

9. To create a realm, click the Master drop-down. In this realm, administrators provide access to users and applications.

   

10. In the Add Realm window, enter a name for the realm and set the parameter Enabled to ON and click Create:

    
    Note

    The realm name is case-sensitive.

11. In the Realm Settings tab, set the following parameters and click Save:

    1. Enabled - ON
    2. User-Managed Access - ON

    3. Copy the link address of SAML 2.0 Identity Provider Metadata

       

12. In the Clients tab, click Create:

    

13. In the Add Client window, set the following parameters and click Save:

    1. Client ID - BASE_URL:8443/auth/saml2/metadata

       Example

       https://magna082.ceph.redhat.com:8443/auth/saml2/metadata

    2. Client Protocol - saml

       

14. In the Clients window, under Settings tab, set the following parameters and click Save:

    1. Client ID - BASE_URL:8443/auth/saml2/metadata

       Example

       https://magna082.ceph.redhat.com:8443/auth/saml2/metadata

    2. Enabled - ON
    3. Client Protocol - saml
    4. Include AuthnStatement - ON
    5. Sign Documents - ON
    6. Signature Algorithm - RSA_SHA1
    7. SAML Signature Key Name - KEY_ID

    8. Valid Redirect URLs - BASE_URL:8443/*

       Example

       https://magna082.ceph.redhat.com:8443/*

    9. Base URL - BASE_URL:8443

       Example

       https://magna082.ceph.redhat.com:8443/

    10. Master SAML Processing URL - http://localhost:8080/auth/realms/REALM_NAME/protocol/saml/descriptor

        Example

        http://localhost:8080/auth/realms/Ceph_LDAP/protocol/saml/descriptor

        Note

        Paste the link of SAML 2.0 Identity Provider Metadata from Realm Settings tab.

        Under Fine Grain SAML Endpoint Configuration, set the parameters:

    11. Assertion Consumer Service POST Binding URL - BASE_URL:8443/#/dashboard

        Example

        https://magna082.ceph.redhat.com:8443/#/dashboard

    12. Assertion Consumer Service Redirect Binding URL - BASE_URL:8443/#/dashboard

        Example

        https://magna082.ceph.redhat.com:8443/#/dashboard

    13. Logout Service Redirect Binding URL - BASE_URL:8443/

        Example

        https://magna082.ceph.redhat.com:8443/

        
        

15. In the Clients window, Mappers tab, set the following parameters and click Save:

    1. Protocol - saml
    2. Name - username
    3. Mapper Property - User Property
    4. Property - username

    5. SAML Attribute name - username

       

16. In the Clients Scope tab, select role_list:

    1. In Mappers tab, select role list, set the Single Role Attribute to ON.

       

17. Select User_Federation tab:

    1. In User Federation window, select ldap from the drop-down:

       

18. In User_Federation window, Settings tab, set the following parameters and click Save:

    1. Console Display Name - rh-ldap
    2. Import Users - ON
    3. Edit_Mode - READ_ONLY
    4. Username LDAP attribute - username
    5. RDN LDAP attribute - username
    6. UUID LDAP attribute - nsuniqueid
    7. User Object Classes - inetOrgPerson, organizationalPerson, rhatPerson

    8. Connection URL - ldap:://myldap.example.com

       Example

       ldap://ldap.corp.redhat.com

       Click Test Connection.

       

       You will get a notification that the LDAP connection is successful.

    9. Users DN - ou=users, dc=example, dc=com

       Example

       ou=users,dc=redhat,dc=com

    10. Bind Type - simple

        
        

    11. Click Test authentication.

        

        You will get a notification that the LDAP authentication is successful.

19. In Mappers tab, select first name row and edit the following parameter and Click Save:

    1. LDAP Attribute - givenName

       
       

20. In User_Federation tab, Settings tab, Click Synchronize all users:

    

    You will get a notification that the sync of users are updated successfully.

    

21. In the Users tab, search for the user added to the dashboard and click the Search icon:

    

22. To view the user , click it’s row. You should see the federation link as the name provided for the User Federation.

    
    Important

    Do not add users manually. If added manually, delete the user by clicking Delete.

23. Users added to the realm and the dashboard can access the Ceph dashboard with their mail address and password.

    Example

    https://magna082.ceph.redhat.com:8443

    

Procedure

1. To configure SSO on Ceph Dashboard, run the following command:

   1. Bare-metal deployments:

      Syntax

      ceph dashboard sso setup saml2 CEPH_DASHBOARD_BASE_URL IDP_METADATA IDP_USERNAME_ATTRIBUTE IDP_ENTITY_ID SP_X_509_CERT SP_PRIVATE_KEY

      Example

      [root@mon ~]# ceph dashboard sso setup saml2 http://dashboard_hostname.ceph.redhat.com:8443 idp-metadata.xml username http://10.70.59.125:8080/auth/realms/realm_name /home/certificate.txt /home/private-key.txt

   2. Container deployments:

      Syntax

      podman exec CEPH_MGR_NODE ceph dashboard sso setup saml2 CEPH_DASHBOARD_BASE_URL IDP_METADATA IDP_USERNAME_ATTRIBUTE IDP_ENTITY_ID SP_X_509_CERT SP_PRIVATE_KEY

      Example

      [root@mon ~]# podman exec ceph-mgr-hostname ceph dashboard sso setup saml2 http://dashboard_hostname.ceph.redhat.com:8443 idp-metadata.xml username http://10.70.59.125:8080/auth/realms/realm_name /home/certificate.txt /home/private-key.txt

   Replace

   • CEPH_MGR_NODE with Ceph mgr node. For example, ceph-mgr-hostname
   • CEPH_DASHBOARD_BASE_URL with the base URL where Ceph Dashboard is accessible.
   • IDP_METADATA with the URL to remote or local path or content of the IdP metadata XML. The supported URL types are http, https, and file.
   • Optional: IDP_USERNAME_ATTRIBUTE with the attribute used to get the username from the authentication response. Defaults to uid.
   • Optional: IDP_ENTITY_ID with the IdP entity ID when more than one entity ID exists on the IdP metadata.
   • Optional: SP_X_509_CERT with the file path of the certificate used by Ceph Dashboard for signing and encryption.
   • Optional: SP_PRIVATE_KEY with the file path of the private key used by Ceph Dashboard for signing and encryption.

2. Verify the current SAML 2.0 configuration:

   1. Bare-metal deployments:

      Syntax

      ceph dashboard sso show saml2

   2. Container deployments:

      Syntax

      podman exec CEPH_MGR_NODE ceph dashboard sso show saml2

3. To enable SSO, run the following command:

   1. Bare-metal deployments:

      Syntax

      ceph dashboard sso enable saml2
      SSO is "enabled" with "SAML2" protocol.

   2. Container deployments:

      Syntax

      podman exec CEPH_MGR_NODE ceph dashboard sso enable saml2
      SSO is "enabled" with "SAML2" protocol.

4. Open your dashboard URL. For example:

   http://dashboard_hostname.ceph.redhat.com:8443

5. On the SSO page, enter the login credentials. SSO redirects to the dashboard web interface.

Procedure

1. To view status of SSO, run the following command:

   1. Bare-metal deployments:

      Syntax

      ceph dashboard sso status
      SSO is "enabled" with "SAML2" protocol.

   2. Container deployments:

      Syntax

      podman exec CEPH_MGR_NODE ceph dashboard sso status
      SSO is "enabled" with "SAML2" protocol.

      Replace

      • CEPH_MGR_NODE with Ceph mgr node. For example, ceph-mgr-hostname

2. To disable SSO, run the following command:

   1. Bare-metal deployments:

      Syntax

      ceph dashboard sso disable
      SSO is "disabled".

   2. Container deployments:

      Syntax

      podman exec CEPH_MGR_NODE ceph dashboard sso disable
      SSO is "disabled".

      Replace

      • CEPH_MGR_NODE with Ceph mgr node. For example, ceph-mgr-hostname

Procedure

1. Log in to the Dashboard.

2. On the upper right side of the Dashboard, click the gear icon and select User management:

   

3. On Roles tab, click the Create button:

   

4. In the CreateRole window, set the Name and select the Permissions for this role, and then click the CreateRole button:

   

5. A notification towards the top right corner of the page indicates the role was created successfully.

   

Procedure

1. Log in to the Dashboard.

2. On the upper right side of the Dashboard, click the gear icon and select User management:

   

3. To view the details of the role, click the row:

   

4. You can see the details of the permissions provided for the roles.

   

Procedure

1. Log in to the Dashboard.

2. On the upper right side of the Dashboard, click the gear icon and select User management:

   

3. To edit the role, click the row:

   

4. On the Roles tab, select Edit from the Edit dropdown menu:

   

5. In the EditRole window, edit parameters including, and then click the EditRole button:

   

6. A notification towards the top right corner of the page indicates the role was updated successfully.

   

Procedure

1. Log in to the Dashboard.

2. On the upper right side of the Dashboard, click the gear icon and select User management:

   

3. To delete the role, click the row:

   

4. On Roles tab, select Delete from the Edit dropdown menu:

   

5. In the Delete Role dialog window, Click the Yes, I am sure box and then Click Delete Role to save the settings:

   

Procedure

1. Log in to the Dashboard.

2. On the upper right side of the Dashboard, click the gear icon and select User management:

   

3. On Users tab, click the Create button:

   

4. In the CreateUser window, set the Username and other parameters including the roles, and then click the _CreateUser_button:

   

5. A notification towards the top right corner of the page indicates the user was created successfully.

   

Procedure

1. Log in to the Dashboard.

2. On the upper right side of the Dashboard, click the gear icon and select User management:

   

3. To edit the user, click the row:

   

4. On Users tab, select Edit from the Edit dropdown menu:

   

5. In the EditUser window, edit parameters including, and then click the EditUser button:

   

6. A notification towards the top right corner of the page indicates the user was updated successfully.

   

Procedure

1. Log in to the Dashboard.

2. On the upper right side of the Dashboard, click the gear icon and select User management:

   

3. To delete the user, click the row:

   

4. On Users tab, select Delete from the Edit dropdown menu:

   

5. In the Delete User dialog window, Click the Yes, I am sure box and then Click Delete user to save the settings:

   

Procedure

1. Log in to the Dashboard:

   

2. After you log in to the dashboard, the landing page loads:

   

3. To return to the landing page after viewing other dashboard pages, click Dashboard towards the top left corner of the page:

   

1. Status
2. Capacity
3. Performance

Procedure

1. Log in to the Dashboard.
2. On the navigation bar, click Cluster and then click Monitors.

3. The Monitors overview page displays information about the overall monitor status as well as tables of in Quorum and Not in quorum Monitor nodes:

   

4. To see the number of open sessions, hover the cursor over the blue dotted trail:

   

5. To see performance counters for any monitor, click its host name:

   

6. View the performance counters:

   

Procedure

1. Log in to the Dashboard.
2. On the navigation bar, click Cluster and then click Hosts.

3. To view the list of hosts, click the Hosts List tab:

   

4. To view the Performance Details of a host, in the Host tab, click its row and select the time range from the Grafana Time Picker drop-down menu:

   

5. To view the performance counters for a specific service on a host, click the service:

   

6. View the performance counters:

   

7. To view performance data for all the hosts:

   1. Click the Overall Performance tab towards the top left of the page:

      

   2. Select the time range from the Grafana Time Picker drop-down:

      

   3. View the Overall Performance page:

      

Procedure

1. Log in to the Dashboard.
2. On the navigation bar, click Cluster and then click OSDs.

3. To view the list of OSDs, click the OSDs List tab:

   

4. To view the attributes of an OSD, on the OSDS List tab, click its row:

   

5. To view the metadata of the OSD, in the OSDs tab, click its row and click the Metadata tab:

   

6. To view the performance counter of the OSD, in the OSDs tab, click its row and click the Performance counter tab:

   

7. To view the histogram of the OSD, in the OSDs tab, click its row and click the Histogram tab:

   

8. To view the performance details of the OSD:

   1. In the OSDs tab, click its row, click the Performance Details tab:

      

   2. Select the time range from the Grafana Time Picker drop-down menu:

      

   3. View the performance details of the OSD:

      

9. To view the overall performance of all the OSDs:

   1. Click the Overall Performance tab towards the top left of the page:

      

   2. Select the time range from the Grafana Time Picker drop-down:

      

10. View the Overall Performance page:

    

Procedure

1. Log in to the Dashboard.

2. On the navigation bar, click Pools:

   

3. View the pools list:

   

4. Mouse over a usage bar graph to view the actual used and free space:

   

5. To view more information about a pool, select it by clicking on its row:

   

6. View the details of the pool. To view performance details and configuration data for the pool, click on the associated tabs.

   

7. To view performance data for all the pools, click the Overall Performance tab towards the top left of the page:

   

8. View the Overall Performance page:

   

Procedure

1. Log in to the dashboard.

2. On the navigation bar, click Filesystems.

   

3. In the example below, you can see the cephfs file system.

   

4. To view details for the file system, click the row for cephfs.

   

5. On the Details tab, you can see metadata servers and their rank plus any standby daemons, at 1, pools and their usage, at 2, and performance counters at 3.

   

6. To view the list of clients which have mounted the file system, click the Clients tab.

   

7. In the example below, you can see the jb-ceph4-client host has the cephfs file system opened.

   

8. To view the performance of the file system, click the Performance Details tab.

   

9. In the example below, you can see the read and write operations, at 1, client requests, at 2, and you can change the time range at 3.

   

Procedure

1. Log in to the Dashboard.

2. On the navigation bar, click Block:

   

3. Select Images from the drop-down:

   

4. Select the Overall Performance tab:

   

5. Select the time range from the Grafana Time Picker drop-down:

   

6. View the Overall Performance page:

   

Procedure

1. Log in to the Dashboard.
2. On the navigation bar, click Cluster.

3. Click CRUSH map.

   

   In the above example, you can see the default CRUSH map, three nodes, and OSDs running on two of the three nodes.

4. Click on the CRUSH map name, nodes, or OSDs, to view details about each object.

   

   In the above example, you can see the values of variables related to an OSD running on the jb-rhel-osd3 node. In particular, note the id is 2.

Procedure

1. Log in to the Dashboard.
2. On the navigation bar, click Cluster.

3. Click Manager modules:

   

   The above screenshot shows the first of three pages of manager modules.

4. Click on a row for a module you want to configure:

   

   Not all modules have configurable parameters. If a module is not configurable, the Edit button is disabled.

5. Towards the upper left of the page, click the Edit button to load the page with the configurable parameters.

   

   The above screenshot shows parameters that you can change for the balancer module. To display a description of a parameter, click the question mark button.

6. To change a parameter, modify the state of the parameter and click the Update button at the bottom of the page:

   

   A notification confirming the change appears in the upper-right corner of the page:

   

Procedure

1. Log in to the Dashboard.
2. Click the Cluster drop-down menu in the top navigation bar.
3. Click Logs in the drop-down menu.

4. View the last thirty unfiltered log entries.

   

   1. To filter by priority, click the Priority drop-down menu and select either Info, Warning, or Error. The example below only shows log entries with the priority of Error.

      

   2. To filter by keyword, enter text into the Keyword form. The example below only shows log entries that include the text osd.2.

      

   3. To filter by date, click the Date form and either use the date picker to select a date from the menu, or enter a date in the form of YYYY-MM-DD. The example below only shows log entries with the date of 2019-10-15.

      

   4. To filter by time, enter a range in the Time range fields using the HH:MM - HH:MM format. Hours must be entered using numbers 0 to 23. The example below only shows log entries from 12:14 to 12:23.

      

   5. To combine filters, set two or more filters. The example below only shows entries that have both a Priority of Warning and the keyword of osd.

      

Procedure

1. Log in to the dashboard.
2. Click the Cluster drop-down menu in the top navigation bar.
3. Click OSDs in the drop-down menu.

4. Click the Cluster-Wide Flags drop-down menu.

   

5. Select Cluster-wide Recovery Priority in the drop-down.

   

6. Optional: Select Priority in the drop-down menu , and then click the Submit button.

   Note

   There are 3 predefined options: Low, Default, High

   

7. Optional: Click Customize priority values, make the required changes, and then click the Submit button.

   

8. A notification towards the top right corner of the page pops up indicating the flags were updated successfully.

   

Procedure

1. Log in to the Dashboard.
2. On the navigation bar, click Cluster and then click Configuration.

3. To view the details of the configuration, click its row:

   

   • You can search for the configuration using the Search box:

     

   • You can filter for the configuration using Level, Service or Source drop-down:

     

4. To edit a configuration, click its row and click the Edit button:

   

   1. In the dialog window, edit the required parameters and Click the Save button:

      

      A notification confirming the change appears in the upper-right corner of the page.

      

Procedure

1. Open and edit the /etc/prometheus/prometheus.yml file.

   1. Under the global section, set the scrape_interval and evaluation_interval options to 15 seconds.

      Example

      global:
        scrape_interval:     15s
        evaluation_interval: 15s

   2. Under the scrape_configs section, add the honor_labels: true option, and edit the targets, and instance options for each of the ceph-mgr nodes.

      Example

      scrape_configs:
        - job_name: 'node'
          honor_labels: true
          static_configs:
          - targets: [ 'node1.example.com:9100' ]
            labels:
              instance: "node1.example.com"
          - targets: ['node2.example.com:9100']
            labels:
              instance: "node2.example.com"

      Note

      Using the honor_labels option enables Ceph to output properly-labelled data relating to any node in the Ceph storage cluster. This allows Ceph to export the proper instance label without Prometheus overwriting it.

   3. To add a new node, simply add the targets, and instance options in the following format:

      Example

      - targets: [ 'new-node.example.com:9100' ]
        labels:
          instance: "new-node"

      Note

      The instance label has to match what appears in Ceph’s OSD metadata instance field, which is the short host name of the node. This helps to correlate Ceph stats with the node’s stats.

2. Add Ceph targets to the /etc/prometheus/ceph_targets.yml file in the following format.

   Example

   [
       {
           "targets": [ "cephnode1.example.com:9283" ],
           "labels": {}
       }
   ]

3. Enable the Prometheus module:

   [root@mon ~]# ceph mgr module enable prometheus

Procedure

1. Take the back-up of the Grafana database:

   1. On the grafana-server node, stop the Grafana service:

      Example

      [root@node04 ~]# systemctl stop grafana-server.service
      [root@node04 ~]# systemctl status grafana-server.service

   2. Take the back-up of the Grafana database:

      Example

      [root@node04 ~]# cp /var/lib/grafana/grafana.db /var/lib/grafana/grafana_backup.db

   3. On the grafana-server node, restart the Grafana service:

      Example

      [root@node04 ~]# systemctl restart grafana-server.service

2. Restore the grafana-server:

   1. On the grafana-server node, if the Grafana service is running, stop the service:

      Example

      [root@node04 ~]# systemctl stop grafana-server.service
      [root@node04 ~]# systemctl status grafana-server.service

   2. Move the backed-up grafana.db file to /var/lib/grafana/ directory:

      Example

      [root@node04 ~]# mv /var/lib/grafana/grafana_backup.db /var/lib/grafana/

   3. On the grafana-server node, restart the Grafana service:

      Example

      [root@node04 ~]# systemctl restart grafana-server.service

3. For the Prometheus alerts, you have to take external back-up of prometheus_data_dir directory, a Ceph-Ansible setting which by default is var/lib/prometheus directory and restore the service using the backed-up directory.

   1. On the grafana-server node, stop the Prometheus service:

      Example

      [root@node04 ~]# systemctl stop prometheus.service
      [root@node04 ~]# systemctl status prometheus.service

   2. Take the back-up of the default Prometheus directory:

      Example

      [root@node04 ~]# cp /var/lib/prometheus/ /var/lib/prometheus_backup/

   3. Replace the prometheus_data_dir directory with the backed-up directory:

      Example

      [root@node04 ~]# mv /var/lib/prometheus_backup/ /var/lib/prometheus_data_dir

   4. On the grafana-server node, restart the prometheus service:

      Example

      [root@node04 ~]# systemctl restart prometheus.service
      [root@node04 ~]# systemctl status prometheus.service

      Note

      If you have made changes to the Prometheus parameters in group_vars/all.yml file, then you have to rerun the playbook.

4. Optional: If the changes do not reflect on the Red Hat Ceph Storage Dashboard, then you have to disable and then enable the dashboard:

   Example

   [root@node04 ~]# ceph mgr module disable dashboard
   [root@node04 ~]# ceph mgr module enable dashboard

Procedure

1. Obtain the access_key and secret_key of an RGW user with the system flag enabled:

   • If you do not have an RGW user with the system flag enabled, create one.

     radosgw-admin user create --uid=USER_ID --display-name=DISPLAY_NAME --system

     Example:

     [root@mon ~]# radosgw-admin user create --uid=rgw-user --display-name=RGW-user --system
     {
         "user_id": "rgw-user",
         "display_name": "RGW-user",
         "email": "",
         "suspended": 0,
         "max_buckets": 1000,
         "subusers": [],
         "keys": [
             {
                 "user": "rgw-user",
                 "access_key": "BYC5SWQQH24A2BFHS2RC",
                 "secret_key": "159d94uHK9ADiWZrGsNYWYjRXCDrhL2xVi8PO6kT"
             }
         ],
         "swift_keys": [],
         "caps": [],
         "op_mask": "read, write, delete",
         "system": "true",
         "default_placement": "",
         "default_storage_class": "",
         "placement_tags": [],
         "bucket_quota": {
             "enabled": false,
             "check_on_raw": false,
             "max_size": -1,
             "max_size_kb": 0,
             "max_objects": -1
         },
         "user_quota": {
             "enabled": false,
             "check_on_raw": false,
             "max_size": -1,
             "max_size_kb": 0,
             "max_objects": -1
         },
         "temp_url_keys": [],
         "type": "rgw",
         "mfa_ids": []
     }

     Take note of the values for access_key and secret_key. In the example above, access_key is BYC5SWQQH24A2BFHS2RC and secret_key is 159d94uHK9ADiWZrGsNYWYjRXCDrhL2xVi8PO6kT.

   • If an RGW user with the system flag enabled is already created, obtain the credentials using the user info command of the radosgw-admin utility.

     radosgw-admin user info --uid=USER_ID

     Example:

     [root@mon ~]# radosgw-admin user info --uid=rgw-user
     {
         "user_id": "rgw-user",
         "display_name": "RGW-user",
         "email": "",
         "suspended": 0,
         "max_buckets": 1000,
         "subusers": [],
         "keys": [
             {
                 "user": "rgw-user",
                 "access_key": "BYC5SWQQH24A2BFHS2RC",
                 "secret_key": "159d94uHK9ADiWZrGsNYWYjRXCDrhL2xVi8PO6kT"
             }
         ],
         "swift_keys": [],
         "caps": [],
         "op_mask": "read, write, delete",
         "system": "true",
         "default_placement": "",
         "default_storage_class": "",
         "placement_tags": [],
         "bucket_quota": {
             "enabled": false,
             "check_on_raw": false,
             "max_size": -1,
             "max_size_kb": 0,
             "max_objects": -1
         },
         "user_quota": {
             "enabled": false,
             "check_on_raw": false,
             "max_size": -1,
             "max_size_kb": 0,
             "max_objects": -1
         },
         "temp_url_keys": [],
         "type": "rgw",
         "mfa_ids": []
     }

     Take note of the values for access_key and secret_key. In the example above, access_key is BYC5SWQQH24A2BFHS2RC and secret_key is 159d94uHK9ADiWZrGsNYWYjRXCDrhL2xVi8PO6kT.

2. Provide the access_key and secret_key credentials to the dashboard:

   1. Provide the access_key to the dashboard.

      ceph dashboard set-rgw-api-access-key ACCESS_KEY

      Example:

      [root@mon ~]# ceph dashboard set-rgw-api-access-key BYC5SWQQH24A2BFHS2RC
      Option RGW_API_ACCESS_KEY updated

   2. Provide the secret_key to the dashboard.

      ceph dashboard set-rgw-api-secret-key SECRET_KEY

      Example:

      [root@mon ~]# ceph dashboard set-rgw-api-secret-key 159d94uHK9ADiWZrGsNYWYjRXCDrhL2xVi8PO6kT
      Option RGW_API_SECRET_KEY updated

3. Provide the host name and port of the object gateway to the dashboard:

   1. Provide the host name to the dashboard.

      ceph dashboard set-rgw-api-host HOST_NAME

      Example:

      [root@mon ~]# ceph dashboard set-rgw-api-host 192.168.122.193
      Option RGW_API_HOST updated

   2. Provide the port to the dashboard.

      ceph dashboard set-rgw-api-port PORT

      Example:

      [root@mon ~]# ceph dashboard set-rgw-api-port 8080
      Option RGW_API_PORT updated

4. Optional: If you are using HTTPS with a self-signed certificate, disable certificate verification in the dashboard to avoid refused connections.

   Refused connections can happen when the certificate is signed by an unknown Certificate Authority, or if the host name used does not match the host name in the certificate.

   ceph dashboard set-rgw-api-ssl-verify false

   Example:

   [root@mon ~]# ceph dashboard set-rgw-api-ssl-verify false
   Option RGW_API_SSL_VERIFY updated