Chapter 12. Configuration of SNMP traps
As a storage administrator, you can deploy and configure the simple network management protocol (SNMP) gateway in a Red Hat Ceph Storage cluster to receive alerts from the Prometheus Alertmanager and route them as SNMP traps to the cluster.
12.1. Simple network management protocol
Simple network management protocol (SNMP) is one of the most widely used open protocols, to monitor distributed systems and devices across a variety of hardware and software platforms. Ceph’s SNMP integration focuses on forwarding alerts from its Prometheus Alertmanager cluster to a gateway daemon. The gateway daemon transforms the alert into an SNMP Notification and sends it on to a designated SNMP management platform. The gateway daemon is from the snmp_notifier_project
, which provides SNMP V2c and V3 support with authentication and encryption.
The Red Hat Ceph Storage SNMP gateway service deploys one instance of the gateway by default. You can increase this by providing placement information. However, if you enable multiple SNMP gateway daemons, your SNMP management platform receives multiple notifications for the same event.
The SNMP traps are alert messages and the Prometheus Alertmanager sends these alerts to the SNMP notifier which then looks for object identifier (OID) in the given alerts’ labels. Each SNMP trap has a unique ID which allows it to send additional traps with updated status to a given SNMP poller. SNMP hooks into the Ceph health checks so that every health warning generates a specific SNMP trap.
In order to work correctly and transfer information on device status to the user to monitor, SNMP relies on several components. There are four main components that makeup SNMP:
- SNMP Manager- The SNMP manager, also called a management station, is a computer that runs network monitoring platforms. A platform that has the job of polling SNMP-enabled devices and retrieving data from them. An SNMP Manager queries agents, receives responses from agents and acknowledges asynchronous events from agents.
- SNMP Agent - An SNMP agent is a program that runs on a system to be managed and contains the MIB database for the system. These collect data like bandwidth and disk space, aggregates it, and sends it to the management information base (MIB).
- Management information base (MIB) - These are components contained within the SNMP agents. The SNMP manager uses this as a database and asks the agent for access to particular information. This information is needed for the network management systems (NMS). The NMS polls the agent to take information from these files and then proceeds to translate it into graphs and displays that can be viewed by the user. MIBs contain statistical and control values that are determined by the network device.
- SNMP Devices
The following versions of SNMP are compatible and supported for gateway implementation:
- V2c - Uses a community string without any authentication and is vulnerable to outside attacks.
- V3 authNoPriv - Uses the username and password authentication without encryption.
- V3 authPriv - Uses the username and password authentication with encryption to the SNMP management platform.
When using SNMP traps, ensure that you have the correct security configuration for your version number to minimize the vulnerabilities that are inherent to SNMP and keep your network protected from unauthorized users.
12.2. Configuring snmptrapd
It is important to configure the simple network management protocol (SNMP) target before deploying the snmp-gateway
because the snmptrapd
daemon contains the auth settings that you need to specify when creating the snmp-gateway
service.
The SNMP gateway feature provides a means of exposing the alerts that are generated in the Prometheus stack to an SNMP management platform. You can configure the SNMP traps to the destination based on the snmptrapd
tool. This tool allows you to establish one or more SNMP trap listeners.
The following parameters are important for configuration:
-
The
engine-id
is a unique identifier for the device, in hex, and required for SNMPV3 gateway. Red Hat recommends using `8000C53F_CLUSTER_FSID_WITHOUT_DASHES_`for this parameter. -
The
snmp-community
, which is the SNMP_COMMUNITY_FOR_SNMPV2 parameter, ispublic
for SNMPV2c gateway. -
The
auth-protocol
which is the AUTH_PROTOCOL, is mandatory for SNMPV3 gateway and isSHA
by default. -
The
privacy-protocol
, which is the PRIVACY_PROTOCOL, is mandatory for SNMPV3 gateway. - The PRIVACY_PASSWORD is mandatory for SNMPV3 gateway with encryption.
- The SNMP_V3_AUTH_USER_NAME is the user name and is mandatory for SNMPV3 gateway.
- The SNMP_V3_AUTH_PASSWORD is the password and is mandatory for SNMPV3 gateway.
Prerequisites
- A running Red Hat Ceph Storage cluster.
- Root-level access to the nodes.
-
Install
firewalld
on Red Hat Enterprise Linux system.
Procedure
On the SNMP management host, install the SNMP packages:
Example
[root@host01 ~]# dnf install -y net-snmp-utils net-snmp
Open the port 162 for SNMP to receive alerts:
Example
[root@host01 ~]# firewall-cmd --zone=public --add-port=162/udp [root@host01 ~]# firewall-cmd --zone=public --add-port=162/udp --permanent
Implement the management information base (MIB) to make sense of the SNMP notification and enhance SNMP support on the destination host. Copy the raw file from the main repository: https://github.com/ceph/ceph/blob/master/monitoring/snmp/CEPH-MIB.txt
Example
[root@host01 ~]# curl -o CEPH_MIB.txt -L https://raw.githubusercontent.com/ceph/ceph/master/monitoring/snmp/CEPH-MIB.txt [root@host01 ~]# scp CEPH_MIB.txt root@host02:/usr/share/snmp/mibs
Create the
snmptrapd
directory.Example
[root@host01 ~]# mkdir /root/snmptrapd/
Create the configuration files in
snmptrapd
directory for each protocol based on the SNMP version:Syntax
format2 %V\n% Agent Address: %A \n Agent Hostname: %B \n Date: %H - %J - %K - %L - %M - %Y \n Enterprise OID: %N \n Trap Type: %W \n Trap Sub-Type: %q \n Community/Infosec Context: %P \n Uptime: %T \n Description: %W \n PDU Attribute/Value Pair Array:\n%v \n -------------- \n createuser -e 0x_ENGINE_ID_ SNMPV3_AUTH_USER_NAME AUTH_PROTOCOL SNMP_V3_AUTH_PASSWORD PRIVACY_PROTOCOL PRIVACY_PASSWORD authuser log,execute SNMP_V3_AUTH_USER_NAME authCommunity log,execute,net SNMP_COMMUNITY_FOR_SNMPV2
For SNMPV2c, create the
snmptrapd_public.conf
file as follows:Example
format2 %V\n% Agent Address: %A \n Agent Hostname: %B \n Date: %H - %J - %K - %L - %M - %Y \n Enterprise OID: %N \n Trap Type: %W \n Trap Sub-Type: %q \n Community/Infosec Context: %P \n Uptime: %T \n Description: %W \n PDU Attribute/Value Pair Array:\n%v \n -------------- \n authCommunity log,execute,net public
The
public
setting here must match thesnmp_community
setting used when deploying thesnmp-gateway
service.For SNMPV3 with authentication only, create the
snmptrapd_auth.conf
file as follows:Example
format2 %V\n% Agent Address: %A \n Agent Hostname: %B \n Date: %H - %J - %K - %L - %M - %Y \n Enterprise OID: %N \n Trap Type: %W \n Trap Sub-Type: %q \n Community/Infosec Context: %P \n Uptime: %T \n Description: %W \n PDU Attribute/Value Pair Array:\n%v \n -------------- \n createuser -e 0x8000C53Ff64f341c655d11eb8778fa163e914bcc myuser SHA mypassword authuser log,execute myuser
The
0x8000C53Ff64f341c655d11eb8778fa163e914bcc
string is theengine_id
, andmyuser
andmypassword
are the credentials. The password security is defined by theSHA
algorithm.This corresponds to the settings for deploying the
snmp-gateway
daemon.Example
snmp_v3_auth_username: myuser snmp_v3_auth_password: mypassword
For SNMPV3 with authentication and encryption, create the
snmptrapd_authpriv.conf
file as follows:Example
format2 %V\n% Agent Address: %A \n Agent Hostname: %B \n Date: %H - %J - %K - %L - %M - %Y \n Enterprise OID: %N \n Trap Type: %W \n Trap Sub-Type: %q \n Community/Infosec Context: %P \n Uptime: %T \n Description: %W \n PDU Attribute/Value Pair Array:\n%v \n -------------- \n createuser -e 0x8000C53Ff64f341c655d11eb8778fa163e914bcc myuser SHA mypassword DES mysecret authuser log,execute myuser
The
0x8000C53Ff64f341c655d11eb8778fa163e914bcc
string is theengine_id
, andmyuser
andmypassword
are the credentials. The password security is defined by theSHA
algorithm andDES
is the type of privacy encryption.This corresponds to the settings for deploying the
snmp-gateway
daemon.Example
snmp_v3_auth_username: myuser snmp_v3_auth_password: mypassword snmp_v3_priv_password: mysecret
Run the daemon on the SNMP management host:
Syntax
/usr/sbin/snmptrapd -M /usr/share/snmp/mibs -m CEPH-MIB.txt -f -C -c /root/snmptrapd/CONFIGURATION_FILE -Of -Lo :162
Example
[root@host01 ~]# /usr/sbin/snmptrapd -M /usr/share/snmp/mibs -m CEPH-MIB.txt -f -C -c /root/snmptrapd/snmptrapd_auth.conf -Of -Lo :162
If any alert is triggered on the storage cluster, you can monitor the output on the SNMP management host. Verify the SNMP traps and also the traps decoded by MIB.
Example
NET-SNMP version 5.8 Agent Address: 0.0.0.0 Agent Hostname: <UNKNOWN> Date: 15 - 5 - 12 - 8 - 10 - 4461391 Enterprise OID: . Trap Type: Cold Start Trap Sub-Type: 0 Community/Infosec Context: TRAP2, SNMP v3, user myuser, context Uptime: 0 Description: Cold Start PDU Attribute/Value Pair Array: .iso.org.dod.internet.mgmt.mib-2.1.3.0 = Timeticks: (292276100) 3 days, 19:52:41.00 .iso.org.dod.internet.snmpV2.snmpModules.1.1.4.1.0 = OID: .iso.org.dod.internet.private.enterprises.ceph.cephCluster.cephNotifications.prometheus.promMgr.promMgrPrometheusInactive .iso.org.dod.internet.private.enterprises.ceph.cephCluster.cephNotifications.prometheus.promMgr.promMgrPrometheusInactive.1 = STRING: "1.3.6.1.4.1.50495.1.2.1.6.2[alertname=CephMgrPrometheusModuleInactive]" .iso.org.dod.internet.private.enterprises.ceph.cephCluster.cephNotifications.prometheus.promMgr.promMgrPrometheusInactive.2 = STRING: "critical" .iso.org.dod.internet.private.enterprises.ceph.cephCluster.cephNotifications.prometheus.promMgr.promMgrPrometheusInactive.3 = STRING: "Status: critical - Alert: CephMgrPrometheusModuleInactive Summary: Ceph's mgr/prometheus module is not available Description: The mgr/prometheus module at 10.70.39.243:9283 is unreachable. This could mean that the module has been disabled or the mgr itself is down. Without the mgr/prometheus module metrics and alerts will no longer function. Open a shell to ceph and use 'ceph -s' to determine whether the mgr is active. If the mgr is not active, restart it, otherwise you can check the mgr/prometheus module is loaded with 'ceph mgr module ls' and if it's not listed as enabled, enable it with 'ceph mgr module enable prometheus'"
In the above example, an alert is generated after the Prometheus module is disabled.
Additional Resources
- See the Deploying the SNMP gateway section in the Red Hat Ceph Storage Operations Guide.
12.3. Deploying the SNMP gateway
You can deploy the simple network management protocol (SNMP) gateway using either SNMPV2c or SNMPV3. There are two methods to deploy the SNMP gateway:
- By creating a credentials file.
- By creating one service configuration yaml file with all the details.
You can use the following parameters to deploy the SNMP gateway based on the versions:
-
The
service_type
is thesnmp-gateway
. -
The
service_name
is any user-defined string. -
The
count
is the number of SNMP gateways to be deployed in a storage cluster. -
The
snmp_destination
parameter must be of the format hostname:port. -
The
engine-id
is a unique identifier for the device, in hex, and required for SNMPV3 gateway. Red Hat recommends to use `8000C53F_CLUSTER_FSID_WITHOUT_DASHES_`for this parameter. -
The
snmp_community
parameter ispublic
for SNMPV2c gateway. -
The
auth-protocol
is mandatory for SNMPV3 gateway and isSHA
by default. -
The
privacy-protocol
is mandatory for SNMPV3 gateway with authentication and encryption. -
The port is
9464
by default. -
You must provide a
-i FILENAME
to pass the secrets and passwords to the orchestrator.
Once the SNMP gateway service is deployed or updated, the Prometheus Alertmanager configuration is automatically updated to forward any alert that has an objectidentifier to the SNMP gateway daemon for further processing.
Prerequisites
- A running Red Hat Ceph Storage cluster.
- Root-level access to the nodes.
-
Configuring
snmptrapd
on the destination host, which is the SNMP management host.
Procedure
Log into the Cephadm shell:
Example
[root@host01 ~]# cephadm shell
Create a label for the host on which SNMP gateway needs to be deployed:
Syntax
ceph orch host label add HOSTNAME snmp-gateway
Example
[ceph: root@host01 /]# ceph orch host label add host02 snmp-gateway
Create a credentials file or a service configuration file based on the SNMP version:
For SNMPV2c, create the file as follows:
Example
[ceph: root@host01 /]# cat snmp_creds.yml snmp_community: public
OR
Example
[ceph: root@host01 /]# cat snmp-gateway.yml service_type: snmp-gateway service_name: snmp-gateway placement: count: 1 spec: credentials: snmp_community: public port: 9464 snmp_destination: 192.168.122.73:162 snmp_version: V2c
For SNMPV3 with authentication only, create the file as follows:
Example
[ceph: root@host01 /]# cat snmp_creds.yml snmp_v3_auth_username: myuser snmp_v3_auth_password: mypassword
OR
Example
[ceph: root@host01 /]# cat snmp-gateway.yml service_type: snmp-gateway service_name: snmp-gateway placement: count: 1 spec: credentials: snmp_v3_auth_password: mypassword snmp_v3_auth_username: myuser engine_id: 8000C53Ff64f341c655d11eb8778fa163e914bcc port: 9464 snmp_destination: 192.168.122.1:162 snmp_version: V3
For SNMPV3 with authentication and encryption, create the file as follows:
Example
[ceph: root@host01 /]# cat snmp_creds.yml snmp_v3_auth_username: myuser snmp_v3_auth_password: mypassword snmp_v3_priv_password: mysecret
OR
Example
[ceph: root@host01 /]# cat snmp-gateway.yml service_type: snmp-gateway service_name: snmp-gateway placement: count: 1 spec: credentials: snmp_v3_auth_password: mypassword snmp_v3_auth_username: myuser snmp_v3_priv_password: mysecret engine_id: 8000C53Ff64f341c655d11eb8778fa163e914bcc port: 9464 snmp_destination: 192.168.122.1:162 snmp_version: V3
Run the
ceph orch
command:Syntax
ceph orch apply snmp-gateway --snmp_version=V2c_OR_V3 --destination=SNMP_DESTINATION [--port=PORT_NUMBER]\ [--engine-id=8000C53F_CLUSTER_FSID_WITHOUT_DASHES_] [--auth-protocol=MDS_OR_SHA] [--privacy_protocol=DES_OR_AES] -i FILENAME
OR
Syntax
ceph orch apply -i FILENAME.yml
For SNMPV2c, with the
snmp_creds
file, run theceph orch
command with thesnmp-version
asV2c
:Example
[ceph: root@host01 /]# ceph orch apply snmp-gateway --snmp-version=V2c --destination=192.168.122.73:162 --port=9464 -i snmp_creds.yml
For SNMPV3 with authentication only, with the
snmp_creds
file, run theceph orch
command with thesnmp-version
asV3
andengine-id
:Example
[ceph: root@host01 /]# ceph orch apply snmp-gateway --snmp-version=V3 --engine-id=8000C53Ff64f341c655d11eb8778fa163e914bcc--destination=192.168.122.73:162 -i snmp_creds.yml
For SNMPV3 with authentication and encryption, with the
snmp_creds
file, run theceph orch
command with thesnmp-version
asV3
,privacy-protocol
, andengine-id
:Example
[ceph: root@host01 /]# ceph orch apply snmp-gateway --snmp-version=V3 --engine-id=8000C53Ff64f341c655d11eb8778fa163e914bcc--destination=192.168.122.73:162 --privacy-protocol=AES -i snmp_creds.yml
OR
For all the SNMP versions, with the
snmp-gateway
file, run the following command:Example
[ceph: root@host01 /]# ceph orch apply -i snmp-gateway.yml
Additional Resources
- See the Configuring `snmptrapd` section in the Red Hat Ceph Storage Operations Guide.