Chapter 14. Integrating with Splunk
If you are using Splunk, you can forward alerts from Red Hat Advanced Cluster Security for Kubernetes to Splunk and view the violations, vulnerability detection, and compliance related data from within Splunk.
Currently, Splunk integration is not supported on IBM Power(ppc64le
) and IBM Z(s390x
).
Depending on your use case, you can integrate Red Hat Advanced Cluster Security for Kubernetes with Splunk by using the following ways:
By using an HTTP event collector in Splunk:
- Use the event collector option to forward alerts and audit log data.
By using the Red Hat Advanced Cluster Security for Kubernetes add-on:
- Use the add-on to pull the violations, vulnerability detection, and compliance data into Splunk.
You can use one or both of these integration options to integrate the Red Hat Advanced Cluster Security for Kubernetes with Splunk.
14.1. Using the HTTP event collector
You can forward alerts from Red Hat Advanced Cluster Security for Kubernetes to Splunk by using an HTTP event collector.
To integrate Red Hat Advanced Cluster Security for Kubernetes with Splunk by using the HTTP event collector, follow these steps:
- Add a new HTTP event collector in Splunk and get the token value.
- Use the token value to set up notifications in Red Hat Advanced Cluster Security for Kubernetes.
- Identify policies for which you want to send notifications, and update the notification settings for those policies.
14.1.1. Adding an HTTP event collector in Splunk
Add a new HTTP event collector for your Splunk instance, and get the token.
Procedure
-
In your Splunk dashboard, go to Settings
Add Data. - Click Monitor.
- On the Add Data page, click HTTP Event Collector.
- Enter a Name for the event collector and then click Next >.
- Accept the default Input Settings and click Review >.
- Review the event collector properties and click Submit >.
- Copy the Token Value for the event collector. You need this token value to configure integration with Splunk in Red Hat Advanced Cluster Security for Kubernetes.
14.1.1.1. Enabling HTTP event collector
You must enable HTTP event collector tokens before you can receive events.
Procedure
-
In your Splunk dashboard, go to Settings
Data inputs. - Click HTTP Event Collector.
- Click Global Settings.
- In the dialog that opens, click Enabled and then click Save.
14.1.2. Configuring Splunk integration in Red Hat Advanced Cluster Security for Kubernetes
Create a new Splunk integration in Red Hat Advanced Cluster Security for Kubernetes by using the token value.
Procedure
-
In the RHACS portal, go to Platform Configuration
Integrations. - Scroll down to the Notifier Integrations section and select Splunk.
-
Click New Integration (
add
icon). - Enter a name for Integration Name.
-
Enter your Splunk URL in the HTTP Event Collector URL field. You must specify the port number if it is not
443
for HTTPS or80
for HTTP. You must also add the URL path/services/collector/event
at the end of the URL. For example,https://<splunk-server-path>:8088/services/collector/event
. Enter your token in the HTTP Event Collector Token field.
NoteIf you are using Red Hat Advanced Cluster Security for Kubernetes version 3.0.57 or newer, you can specify custom Source Type for Alert events and Source Type for Audit events.
- Select Test to send a test message to verify that the integration with Splunk is working.
- Select Create to generate the configuration.
14.1.3. Configuring policy notifications
Enable alert notifications for system policies.
Procedure
-
In the RHACS portal, go to Platform Configuration
Policy Management. - Select one or more policies for which you want to send alerts.
- Under Bulk actions, select Enable notification.
In the Enable notification window, select the Splunk notifier.
NoteIf you have not configured any other integrations, the system displays a message that no notifiers are configured.
- Click Enable.
- Red Hat Advanced Cluster Security for Kubernetes sends notifications on an opt-in basis. To receive notifications, you must first assign a notifier to the policy.
- Notifications are only sent once for a given alert. If you have assigned a notifier to a policy, you will not receive a notification unless a violation generates a new alert.
Red Hat Advanced Cluster Security for Kubernetes creates a new alert for the following scenarios:
- A policy violation occurs for the first time in a deployment.
- A runtime-phase policy violation occurs in a deployment after you resolved the previous runtime alert for a policy in that deployment.
14.2. Using the Red Hat Advanced Cluster Security for Kubernetes add-on
You can use the Red Hat Advanced Cluster Security for Kubernetes add-on to forward the vulnerability detection and compliance related data from the Red Hat Advanced Cluster Security for Kubernetes to Splunk.
Generate an API token with read
permission for all resources in Red Hat Advanced Cluster Security for Kubernetes and then use that token to install and configure the add-on.
14.2.1. Installing and configuring the Splunk add-on
You can install the Red Hat Advanced Cluster Security for Kubernetes add-on from your Splunk instance.
To maintain backward compatibility with the StackRox Kubernetes Security Platform add-on, the source_type
and input_type
parameters for configured inputs are still called stackrox_compliance
, stackrox_violations
, and stackrox_vulnerability_management
.
Prerequisites
-
You must have an API token with
read
permission for all resources of Red Hat Advanced Cluster Security for Kubernetes. You can assign the Analyst system role to grant this level of access. The Analyst role has read permissions for all resources.
Procedure
- Download the Red Hat Advanced Cluster Security for Kubernetes add-on from Splunkbase.
- Go to the Splunk home page on your Splunk instance.
-
Go to Apps
Manage Apps. - Select Install app from file.
- In the Upload app pop-up box, select Choose File and select the Red Hat Advanced Cluster Security for Kubernetes add-on file.
- Click Upload.
- Click Restart Splunk, and confirm to restart.
- After Splunk restarts, select Red Hat Advanced Cluster Security for Kubernetes from the Apps menu.
Go to Configuration and then click Add-on Settings.
-
For Central Endpoint, enter the IP address or the name of your Central instance. For example,
central.custom:443
. - Enter the API token you have generated for the add-on.
- Click Save.
-
For Central Endpoint, enter the IP address or the name of your Central instance. For example,
- Go to Inputs.
Click Create New Input, and select one of the following:
- ACS Compliance to pull the compliance data.
- ACS Violations to pull the violations data.
- ACS Vulnerability Management to pull the vulnerabilities data.
- Enter a Name for the input.
- Select an Interval to pull data from Red Hat Advanced Cluster Security for Kubernetes. For example, every 14400 seconds.
- Select the Splunk Index to which you want to send the data.
- For Central Endpoint, enter the IP address or the name of your Central instance.
- Enter the API token you have generated for the add-on.
- Click Add.
Verification
To verify the the Red Hat Advanced Cluster Security for Kubernetes add-on installation, query the received data.
-
In your Splunk instance, go to Search and type
index=* sourcetype="stackrox-*"
as the query. - Press Enter.
-
In your Splunk instance, go to Search and type
Verify that your configured sources are displayed in the search results.
14.2.2. Update the StackRox Kubernetes Security Platform add-on
If you are using the StackRox Kubernetes Security Platform add-on, you must upgrade to the new Red Hat Advanced Cluster Security for Kubernetes add-on.
You can see the update notification on the Splunk homepage under the list of apps on the left. Alternatively, you can also go to the Apps
Prerequisites
-
You must have an API token with
read
permission for all resources of Red Hat Advanced Cluster Security for Kubernetes. You can assign the Analyst system role to grant this level of access. The Analyst role has read permissions for all the resources.
Procedure
- Click Update on the update notification.
- Select the checkbox for accepting the terms and conditions, and then click Accept and Continue to install the update.
- After the installation, select Red Hat Advanced Cluster Security for Kubernetes from the Apps menu.
Go to Configuration and then click Add-on Settings.
- Enter the API token you have generated for the add-on.
- Click Save.
14.2.3. Troubleshoot the Splunk add-on
If you stop receiving events from the Red Hat Advanced Cluster Security for Kubernetes add-on, check the Splunk add-on debug logs for errors.
Splunk creates a debug log file for every configured input in the /opt/splunk/var/log/splunk
directory. Find the file named stackrox_<input>_<uid>.log
, for example, stackrox_compliance_29a3e14798aa2363d.log
and look for issues.