Chapter 13. Monitoring bare-metal events with the Bare Metal Event Relay

Procedure

1. Create a namespace for the Bare Metal Event Relay.

   1. Save the following YAML in the bare-metal-events-namespace.yaml file:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-bare-metal-events
        labels:
          name: openshift-bare-metal-events
          openshift.io/cluster-monitoring: "true"

      Copy to Clipboard Toggle word wrap

   2. Create the Namespace CR:

      $ oc create -f bare-metal-events-namespace.yaml

      Copy to Clipboard Toggle word wrap

2. Create an Operator group for the Bare Metal Event Relay Operator.

   1. Save the following YAML in the bare-metal-events-operatorgroup.yaml file:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: bare-metal-event-relay-group
        namespace: openshift-bare-metal-events
      spec:
        targetNamespaces:
        - openshift-bare-metal-events

      Copy to Clipboard Toggle word wrap

   2. Create the OperatorGroup CR:

      $ oc create -f bare-metal-events-operatorgroup.yaml

      Copy to Clipboard Toggle word wrap

3. Subscribe to the Bare Metal Event Relay.

   1. Save the following YAML in the bare-metal-events-sub.yaml file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: bare-metal-event-relay-subscription
        namespace: openshift-bare-metal-events
      spec:
        channel: "stable"
        name: bare-metal-event-relay
        source: redhat-operators
        sourceNamespace: openshift-marketplace

      Copy to Clipboard Toggle word wrap

   2. Create the Subscription CR:

      $ oc create -f bare-metal-events-sub.yaml

      Copy to Clipboard Toggle word wrap

Procedure

1. Install the Bare Metal Event Relay using the OpenShift Container Platform web console:

   1. In the OpenShift Container Platform web console, click Operators OperatorHub.
   2. Choose Bare Metal Event Relay from the list of available Operators, and then click Install.
   3. On the Install Operator page, select or create a Namespace, select openshift-bare-metal-events, and then click Install.

1. Switch to the Operators Installed Operators page.

2. Ensure that Bare Metal Event Relay is listed in the project with a Status of InstallSucceeded.

   Note

   During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

Procedure

1. Confirm that the node hardware has the Redfish EventService enabled by running the following curl command:

   curl https://<bmc_ip_address>/redfish/v1/EventService --insecure -H 'Content-Type: application/json' -u "<bmc_username>:<password>"

   Copy to Clipboard Toggle word wrap

   where:

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

   Example output

   {
      "@odata.context": "/redfish/v1/$metadata#EventService.EventService",
      "@odata.id": "/redfish/v1/EventService",
      "@odata.type": "#EventService.v1_0_2.EventService",
      "Actions": {
         "#EventService.SubmitTestEvent": {
            "EventType@Redfish.AllowableValues": ["StatusChange", "ResourceUpdated", "ResourceAdded", "ResourceRemoved", "Alert"],
            "target": "/redfish/v1/EventService/Actions/EventService.SubmitTestEvent"
         }
      },
      "DeliveryRetryAttempts": 3,
      "DeliveryRetryIntervalSeconds": 30,
      "Description": "Event Service represents the properties for the service",
      "EventTypesForSubscription": ["StatusChange", "ResourceUpdated", "ResourceAdded", "ResourceRemoved", "Alert"],
      "EventTypesForSubscription@odata.count": 5,
      "Id": "EventService",
      "Name": "Event Service",
      "ServiceEnabled": true,
      "Status": {
         "Health": "OK",
         "HealthRollup": "OK",
         "State": "Enabled"
      },
      "Subscriptions": {
         "@odata.id": "/redfish/v1/EventService/Subscriptions"
      }
   }

   Copy to Clipboard Toggle word wrap

2. Get the Bare Metal Event Relay service route for the cluster by running the following command:

   $ oc get route -n openshift-bare-metal-events

   Copy to Clipboard Toggle word wrap

   Example output

   NAME             HOST/PORT                                                                                           PATH   SERVICES                 PORT   TERMINATION   WILDCARD
   hw-event-proxy   hw-event-proxy-openshift-bare-metal-events.apps.compute-1.example.com          hw-event-proxy-service   9087   edge          None

   Copy to Clipboard Toggle word wrap

3. Create a BMCEventSubscription resource to subscribe to the Redfish events:

   1. Save the following YAML in the bmc_sub.yaml file:

      apiVersion: metal3.io/v1alpha1
      kind: BMCEventSubscription
      metadata:
        name: sub-01
        namespace: openshift-machine-api
      spec:
         hostName: <hostname> 

      1

      
         destination: <proxy_service_url> 

      2

      
         context: ''

      Copy to Clipboard Toggle word wrap

      1

      Specifies the name or UUID of the worker node where the Redfish events are generated.

      2

      Specifies the bare-metal event proxy service, for example, https://hw-event-proxy-openshift-bare-metal-events.apps.compute-1.example.com/webhook.

   2. Create the BMCEventSubscription CR:

      $ oc create -f bmc_sub.yaml

      Copy to Clipboard Toggle word wrap

4. Optional: To delete the BMC event subscription, run the following command:

   $ oc delete -f bmc_sub.yaml

   Copy to Clipboard Toggle word wrap

5. Optional: To manually create a Redfish event subscription without creating a BMCEventSubscription CR, run the following curl command, specifying the BMC username and password.

   $ curl -i -k -X POST -H "Content-Type: application/json"  -d '{"Destination": "https://<proxy_service_url>", "Protocol" : "Redfish", "EventTypes": ["Alert"], "Context": "root"}' -u <bmc_username>:<password> 'https://<bmc_ip_address>/redfish/v1/EventService/Subscriptions' –v

   Copy to Clipboard Toggle word wrap

   where:

   proxy_service_url

   is the bare-metal event proxy service, for example, https://hw-event-proxy-openshift-bare-metal-events.apps.compute-1.example.com/webhook.

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

   Example output

   HTTP/1.1 201 Created
   Server: AMI MegaRAC Redfish Service
   Location: /redfish/v1/EventService/Subscriptions/1
   Allow: GET, POST
   Access-Control-Allow-Origin: *
   Access-Control-Expose-Headers: X-Auth-Token
   Access-Control-Allow-Headers: X-Auth-Token
   Access-Control-Allow-Credentials: true
   Cache-Control: no-cache, must-revalidate
   Link: <http://redfish.dmtf.org/schemas/v1/EventDestination.v1_6_0.json>; rel=describedby
   Link: <http://redfish.dmtf.org/schemas/v1/EventDestination.v1_6_0.json>
   Link: </redfish/v1/EventService/Subscriptions>; path=
   ETag: "1651135676"
   Content-Type: application/json; charset=UTF-8
   OData-Version: 4.0
   Content-Length: 614
   Date: Thu, 28 Apr 2022 08:47:57 GMT

   Copy to Clipboard Toggle word wrap

Procedure

1. Check the current subscriptions for the BMC by running the following curl command:

   $ curl --globoff -H "Content-Type: application/json" -k -X GET --user <bmc_username>:<password> https://<bmc_ip_address>/redfish/v1/EventService/Subscriptions

   Copy to Clipboard Toggle word wrap

   where:

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

   Example output

   % Total % Received % Xferd Average Speed Time Time Time Current
   Dload Upload Total Spent Left Speed
   100 435 100 435 0 0 399 0 0:00:01 0:00:01 --:--:-- 399
   {
     "@odata.context": "/redfish/v1/$metadata#EventDestinationCollection.EventDestinationCollection",
     "@odata.etag": ""
     1651137375 "",
     "@odata.id": "/redfish/v1/EventService/Subscriptions",
     "@odata.type": "#EventDestinationCollection.EventDestinationCollection",
     "Description": "Collection for Event Subscriptions",
     "Members": [
     {
       "@odata.id": "/redfish/v1/EventService/Subscriptions/1"
     }],
     "Members@odata.count": 1,
     "Name": "Event Subscriptions Collection"
   }

   Copy to Clipboard Toggle word wrap

   In this example, a single subscription is configured: /redfish/v1/EventService/Subscriptions/1.

2. Optional: To remove the /redfish/v1/EventService/Subscriptions/1 subscription with curl, run the following command, specifying the BMC username and password:

   $ curl --globoff -L -w "%{http_code} %{url_effective}\n" -k -u <bmc_username>:<password >-H "Content-Type: application/json" -d '{}' -X DELETE https://<bmc_ip_address>/redfish/v1/EventService/Subscriptions/1

   Copy to Clipboard Toggle word wrap

   where:

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

Procedure

1. Create the HardwareEvent custom resource (CR):

   1. Save the following YAML in the hw-event.yaml file:

      apiVersion: "event.redhat-cne.org/v1alpha1"
      kind: "HardwareEvent"
      metadata:
        name: "hardware-event"
      spec:
        nodeSelector:
          node-role.kubernetes.io/hw-event: "" 

      1

      
        transportHost: "amqp://amq-router-service-name.amq-namespace.svc.cluster.local" 

      2

      
        logLevel: "debug" 

      3

      
        msgParserTimeout: "10" 

      4

      Copy to Clipboard Toggle word wrap

      1

      Required. Use the nodeSelector field to target nodes with the specified label, for example, node-role.kubernetes.io/hw-event: "".

      2

      Required. AMQP host that delivers the events at the transport layer using the AMQP protocol.

      3

      Optional. The default value is debug. Sets the log level in hw-event-proxy logs. The following log levels are available: fatal, error, warning, info, debug, trace.

      4

      Optional. Sets the timeout value in milliseconds for the Message Parser. If a message parsing request is not responded to within the timeout duration, the original hardware event message is passed to the cloud native event framework. The default value is 10.

   2. Create the HardwareEvent CR:

      $ oc create -f hardware-event.yaml

      Copy to Clipboard Toggle word wrap

2. Create a BMC username and password Secret CR that enables the hardware events proxy to access the Redfish message registry for the bare-metal host.

   1. Save the following YAML in the hw-event-bmc-secret.yaml file:

      apiVersion: v1
      kind: Secret
      metadata:
        name: redfish-basic-auth
      type: Opaque
      stringData: 

      1

      
        username: <bmc_username>
        password: <bmc_password>
        # BMC host DNS or IP address
        hostaddr: <bmc_host_ip_address>

      Copy to Clipboard Toggle word wrap

      1

      Enter plain text values for the various items under stringData.

   2. Create the Secret CR:

      $ oc create -f hw-event-bmc-secret.yaml

      Copy to Clipboard Toggle word wrap