Chapter 10. Viewing cluster logs by using Kibana
The logging subsystem includes a web console for visualizing collected log data. Currently, OpenShift Container Platform deploys the Kibana console for visualization.
Using the log visualizer, you can do the following with your data:
- search and browse the data using the Discover tab.
- chart and map the data using the Visualize tab.
- create and view custom dashboards using the Dashboard tab.
Use and configuration of the Kibana interface is beyond the scope of this documentation. For more information, on using the interface, see the Kibana documentation.
The audit logs are not stored in the internal OpenShift Container Platform Elasticsearch instance by default. To view the audit logs in Kibana, you must use the Log Forwarding API to configure a pipeline that uses the default output for audit logs.
10.1. Defining Kibana index patterns
An index pattern defines the Elasticsearch indices that you want to visualize. To explore and visualize data in Kibana, you must create an index pattern.
Prerequisites
A user must have the
cluster-adminrole, thecluster-readerrole, or both roles to view the infra and audit indices in Kibana. The defaultkubeadminuser has proper permissions to view these indices.If you can view the pods and logs in the
default,kube-andopenshift-projects, you should be able to access these indices. You can use the following command to check if the current user has appropriate permissions:$ oc auth can-i get pods/log -n <project>
Example output
yes
NoteThe audit logs are not stored in the internal OpenShift Container Platform Elasticsearch instance by default. To view the audit logs in Kibana, you must use the Log Forwarding API to configure a pipeline that uses the
defaultoutput for audit logs.- Elasticsearch documents must be indexed before you can create index patterns. This is done automatically, but it might take a few minutes in a new or updated cluster.
Procedure
To define index patterns and create visualizations in Kibana:
-
In the OpenShift Container Platform console, click the Application Launcher
and select Logging.
Create your Kibana index patterns by clicking Management
Index Patterns Create index pattern: -
Each user must manually create index patterns when logging into Kibana the first time to see logs for their projects. Users must create an index pattern named
appand use the@timestamptime field to view their container logs. -
Each admin user must create index patterns when logged into Kibana the first time for the
app,infra, andauditindices using the@timestamptime field.
-
Each user must manually create index patterns when logging into Kibana the first time to see logs for their projects. Users must create an index pattern named
- Create Kibana Visualizations from the new index patterns.
10.2. Viewing cluster logs in Kibana
You view cluster logs in the Kibana web console. The methods for viewing and visualizing your data in Kibana that are beyond the scope of this documentation. For more information, refer to the Kibana documentation.
Prerequisites
- The Red Hat OpenShift Logging and Elasticsearch Operators must be installed.
- Kibana index patterns must exist.
A user must have the
cluster-adminrole, thecluster-readerrole, or both roles to view the infra and audit indices in Kibana. The defaultkubeadminuser has proper permissions to view these indices.If you can view the pods and logs in the
default,kube-andopenshift-projects, you should be able to access these indices. You can use the following command to check if the current user has appropriate permissions:$ oc auth can-i get pods/log -n <project>
Example output
yes
NoteThe audit logs are not stored in the internal OpenShift Container Platform Elasticsearch instance by default. To view the audit logs in Kibana, you must use the Log Forwarding API to configure a pipeline that uses the
defaultoutput for audit logs.
Procedure
To view logs in Kibana:
-
In the OpenShift Container Platform console, click the Application Launcher
and select Logging.
Log in using the same credentials you use to log in to the OpenShift Container Platform console.
The Kibana interface launches.
- In Kibana, click Discover.
Select the index pattern you created from the drop-down menu in the top-left corner: app, audit, or infra.
The log data displays as time-stamped documents.
- Expand one of the time-stamped documents.
Click the JSON tab to display the log entry for that document.
Example 10.1. Sample infrastructure log entry in Kibana
{ "_index": "infra-000001", "_type": "_doc", "_id": "YmJmYTBlNDkZTRmLTliMGQtMjE3NmFiOGUyOWM3", "_version": 1, "_score": null, "_source": { "docker": { "container_id": "f85fa55bbef7bb783f041066be1e7c267a6b88c4603dfce213e32c1" }, "kubernetes": { "container_name": "registry-server", "namespace_name": "openshift-marketplace", "pod_name": "redhat-marketplace-n64gc", "container_image": "registry.redhat.io/redhat/redhat-marketplace-index:v4.7", "container_image_id": "registry.redhat.io/redhat/redhat-marketplace-index@sha256:65fc0c45aabb95809e376feb065771ecda9e5e59cc8b3024c4545c168f", "pod_id": "8f594ea2-c866-4b5c-a1c8-a50756704b2a", "host": "ip-10-0-182-28.us-east-2.compute.internal", "master_url": "https://kubernetes.default.svc", "namespace_id": "3abab127-7669-4eb3-b9ef-44c04ad68d38", "namespace_labels": { "openshift_io/cluster-monitoring": "true" }, "flat_labels": [ "catalogsource_operators_coreos_com/update=redhat-marketplace" ] }, "message": "time=\"2020-09-23T20:47:03Z\" level=info msg=\"serving registry\" database=/database/index.db port=50051", "level": "unknown", "hostname": "ip-10-0-182-28.internal", "pipeline_metadata": { "collector": { "ipaddr4": "10.0.182.28", "inputname": "fluent-plugin-systemd", "name": "fluentd", "received_at": "2020-09-23T20:47:15.007583+00:00", "version": "1.7.4 1.6.0" } }, "@timestamp": "2020-09-23T20:47:03.422465+00:00", "viaq_msg_id": "YmJmYTBlNDktMDMGQtMjE3NmFiOGUyOWM3", "openshift": { "labels": { "logging": "infra" } } }, "fields": { "@timestamp": [ "2020-09-23T20:47:03.422Z" ], "pipeline_metadata.collector.received_at": [ "2020-09-23T20:47:15.007Z" ] }, "sort": [ 1600894023422 ] }
