Chapter 11. Monitoring bare-metal events with the Bare Metal Event Relay

Procedure

1. Create a namespace for the Bare Metal Event Relay.

   1. Save the following YAML in the bare-metal-events-namespace.yaml file:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-bare-metal-events
        labels:
          name: openshift-bare-metal-events
          openshift.io/cluster-monitoring: "true"

   2. Create the Namespace CR:

      $ oc create -f bare-metal-events-namespace.yaml

2. Create an Operator group for the Bare Metal Event Relay Operator.

   1. Save the following YAML in the bare-metal-events-operatorgroup.yaml file:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: bare-metal-event-relay-group
        namespace: openshift-bare-metal-events
      spec:
        targetNamespaces:
        - openshift-bare-metal-events

   2. Create the OperatorGroup CR:

      $ oc create -f bare-metal-events-operatorgroup.yaml

3. Subscribe to the Bare Metal Event Relay.

   1. Save the following YAML in the bare-metal-events-sub.yaml file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: bare-metal-event-relay-subscription
        namespace: openshift-bare-metal-events
      spec:
        channel: "stable"
        name: bare-metal-event-relay
        source: redhat-operators
        sourceNamespace: openshift-marketplace

   2. Create the Subscription CR:

      $ oc create -f bare-metal-events-sub.yaml

Procedure

1. Install the Bare Metal Event Relay using the OpenShift Container Platform web console:

   1. In the OpenShift Container Platform web console, click Operators OperatorHub.
   2. Choose Bare Metal Event Relay from the list of available Operators, and then click Install.
   3. On the Install Operator page, select or create a Namespace, select openshift-bare-metal-events, and then click Install.

1. Switch to the Operators Installed Operators page.

2. Ensure that Bare Metal Event Relay is listed in the project with a Status of InstallSucceeded.

   Note

   During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

Procedure

1. Confirm that the node hardware has the Redfish EventService enabled by running the following curl command:

   curl https://<bmc_ip_address>/redfish/v1/EventService --insecure -H 'Content-Type: application/json' -u "<bmc_username>:<password>"

   where:

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

   Example output

   {
      "@odata.context": "/redfish/v1/$metadata#EventService.EventService",
      "@odata.id": "/redfish/v1/EventService",
      "@odata.type": "#EventService.v1_0_2.EventService",
      "Actions": {
         "#EventService.SubmitTestEvent": {
            "EventType@Redfish.AllowableValues": ["StatusChange", "ResourceUpdated", "ResourceAdded", "ResourceRemoved", "Alert"],
            "target": "/redfish/v1/EventService/Actions/EventService.SubmitTestEvent"
         }
      },
      "DeliveryRetryAttempts": 3,
      "DeliveryRetryIntervalSeconds": 30,
      "Description": "Event Service represents the properties for the service",
      "EventTypesForSubscription": ["StatusChange", "ResourceUpdated", "ResourceAdded", "ResourceRemoved", "Alert"],
      "EventTypesForSubscription@odata.count": 5,
      "Id": "EventService",
      "Name": "Event Service",
      "ServiceEnabled": true,
      "Status": {
         "Health": "OK",
         "HealthRollup": "OK",
         "State": "Enabled"
      },
      "Subscriptions": {
         "@odata.id": "/redfish/v1/EventService/Subscriptions"
      }
   }

2. Get the Bare Metal Event Relay service route for the cluster by running the following command:

   $ oc get route -n openshift-bare-metal-events

   Example output

   NAME             HOST/PORT                                                                                           PATH   SERVICES                 PORT   TERMINATION   WILDCARD
   hw-event-proxy   hw-event-proxy-openshift-bare-metal-events.apps.compute-1.example.com          hw-event-proxy-service   9087   edge          None

3. Create a BMCEventSubscription resource to subscribe to the Redfish events:

   1. Save the following YAML in the bmc_sub.yaml file:

      apiVersion: metal3.io/v1alpha1
      kind: BMCEventSubscription
      metadata:
        name: sub-01
        namespace: openshift-machine-api
      spec:
         hostName: <hostname> 1
         destination: <proxy_service_url> 2
         context: ''

      1

      Specifies the name or UUID of the worker node where the Redfish events are generated.

      2

      Specifies the bare-metal event proxy service, for example, https://hw-event-proxy-openshift-bare-metal-events.apps.compute-1.example.com/webhook.

   2. Create the BMCEventSubscription CR:

      $ oc create -f bmc_sub.yaml

4. Optional: To delete the BMC event subscription, run the following command:

   $ oc delete -f bmc_sub.yaml

5. Optional: To manually create a Redfish event subscription without creating a BMCEventSubscription CR, run the following curl command, specifying the BMC username and password.

   $ curl -i -k -X POST -H "Content-Type: application/json"  -d '{"Destination": "https://<proxy_service_url>", "Protocol" : "Redfish", "EventTypes": ["Alert"], "Context": "root"}' -u <bmc_username>:<password> 'https://<bmc_ip_address>/redfish/v1/EventService/Subscriptions' –v

   where:

   proxy_service_url

   is the bare-metal event proxy service, for example, https://hw-event-proxy-openshift-bare-metal-events.apps.compute-1.example.com/webhook.

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

   Example output

   HTTP/1.1 201 Created
   Server: AMI MegaRAC Redfish Service
   Location: /redfish/v1/EventService/Subscriptions/1
   Allow: GET, POST
   Access-Control-Allow-Origin: *
   Access-Control-Expose-Headers: X-Auth-Token
   Access-Control-Allow-Headers: X-Auth-Token
   Access-Control-Allow-Credentials: true
   Cache-Control: no-cache, must-revalidate
   Link: <http://redfish.dmtf.org/schemas/v1/EventDestination.v1_6_0.json>; rel=describedby
   Link: <http://redfish.dmtf.org/schemas/v1/EventDestination.v1_6_0.json>
   Link: </redfish/v1/EventService/Subscriptions>; path=
   ETag: "1651135676"
   Content-Type: application/json; charset=UTF-8
   OData-Version: 4.0
   Content-Length: 614
   Date: Thu, 28 Apr 2022 08:47:57 GMT

Procedure

1. Check the current subscriptions for the BMC by running the following curl command:

   $ curl --globoff -H "Content-Type: application/json" -k -X GET --user <bmc_username>:<password> https://<bmc_ip_address>/redfish/v1/EventService/Subscriptions

   where:

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

   Example output

   % Total % Received % Xferd Average Speed Time Time Time Current
   Dload Upload Total Spent Left Speed
   100 435 100 435 0 0 399 0 0:00:01 0:00:01 --:--:-- 399
   {
     "@odata.context": "/redfish/v1/$metadata#EventDestinationCollection.EventDestinationCollection",
     "@odata.etag": ""
     1651137375 "",
     "@odata.id": "/redfish/v1/EventService/Subscriptions",
     "@odata.type": "#EventDestinationCollection.EventDestinationCollection",
     "Description": "Collection for Event Subscriptions",
     "Members": [
     {
       "@odata.id": "/redfish/v1/EventService/Subscriptions/1"
     }],
     "Members@odata.count": 1,
     "Name": "Event Subscriptions Collection"
   }

   In this example, a single subscription is configured: /redfish/v1/EventService/Subscriptions/1.

2. Optional: To remove the /redfish/v1/EventService/Subscriptions/1 subscription with curl, run the following command, specifying the BMC username and password:

   $ curl --globoff -L -w "%{http_code} %{url_effective}\n" -k -u <bmc_username>:<password >-H "Content-Type: application/json" -d '{}' -X DELETE https://<bmc_ip_address>/redfish/v1/EventService/Subscriptions/1

   where:

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

Procedure

1. Create the HardwareEvent custom resource (CR):

   1. Save the following YAML in the hw-event.yaml file:

      apiVersion: "event.redhat-cne.org/v1alpha1"
      kind: "HardwareEvent"
      metadata:
        name: "hardware-event"
      spec:
        nodeSelector:
          node-role.kubernetes.io/hw-event: "" 1
        transportHost: "amqp://amq-router-service-name.amq-namespace.svc.cluster.local" 2
        logLevel: "debug" 3
        msgParserTimeout: "10" 4

      1

      Required. Use the nodeSelector field to target nodes with the specified label, for example, node-role.kubernetes.io/hw-event: "".

      2

      Required. AMQP host that delivers the events at the transport layer using the AMQP protocol.

      3

      Optional. The default value is debug. Sets the log level in hw-event-proxy logs. The following log levels are available: fatal, error, warning, info, debug, trace.

      4

      Optional. Sets the timeout value in milliseconds for the Message Parser. If a message parsing request is not responded to within the timeout duration, the original hardware event message is passed to the cloud native event framework. The default value is 10.

   2. Create the HardwareEvent CR:

      $ oc create -f hardware-event.yaml

2. Create a BMC username and password Secret CR that enables the hardware events proxy to access the Redfish message registry for the bare-metal host.

   1. Save the following YAML in the hw-event-bmc-secret.yaml file:

      apiVersion: v1
      kind: Secret
      metadata:
        name: redfish-basic-auth
      type: Opaque
      stringData: 1
        username: <bmc_username>
        password: <bmc_password>
        # BMC host DNS or IP address
        hostaddr: <bmc_host_ip_address>

      1

      Enter plain text values for the various items under stringData.

   2. Create the Secret CR:

      $ oc create -f hw-event-bmc-secret.yaml