Chapter 18. Network Observability CLI
18.1. Installing the Network Observability CLI Copy linkLink copied to clipboard!
The Network Observability CLI (oc netobserv) is a standalone OpenShift CLI (oc) plugin used to debug and troubleshoot cluster network traffic. It operates independently of the Network Observability Operator to gather immediate network performance diagnostics.
18.1.1. Network Observability CLI overview Copy linkLink copied to clipboard!
Use the Network Observability CLI (oc netobserv) to quickly debug and troubleshoot networking issues. This tool provides instant, live insight into flows and packets without installing the Network Observability Operator.
The Network Observability CLI is a flow and packet visualization tool that relies on eBPF agents to stream collected data to an ephemeral collector pod. It requires no persistent storage during the capture. After the run, the output is transferred to your local machine.
CLI capture is meant to run only for short durations, such as 8-10 minutes. If it runs for too long, it can be difficult to delete the running process.
18.1.2. Install the Network Observability CLI Copy linkLink copied to clipboard!
The Network Observability CLI gives you a lightweight way to quickly debug and troubleshoot network observability. It must be installed separately.
Installing the Network Observability CLI (oc netobserv) is a separate procedure from the Network Observability Operator installation. This means that, even if the Operator is installed from the software catalog, the CLI must be installed separately.
Users can optionally use Krew to install the netobserv CLI plugin. For more information, see "Installing a CLI plugin with Krew".
Prerequisites
-
You must install the OpenShift CLI (
oc). - You must have a macOS or Linux operating system.
-
You must install either
dockerorpodman.
You can use podman or docker to run the installation commands. This procedure uses podman.
Procedure
Log in to the Red Hat registry by running the following command:
$ podman login registry.redhat.ioExtract the
oc-netobservfile from the image by running the following commands:$ podman create --name netobserv-cli registry.redhat.io/network-observability/network-observability-cli-rhel9:1.11 $ podman cp netobserv-cli:/oc-netobserv . $ podman rm netobserv-cliMove the extracted file to a directory that is on the system’s
PATH, such as/usr/local/bin/, by running the following command:$ sudo mv oc-netobserv /usr/local/bin/
Verification
Verify that
oc netobservis available:$ oc netobserv versionThis command should produce an outcome similar to the following example:
Netobserv CLI version <version>
18.2. Using the Network Observability CLI Copy linkLink copied to clipboard!
The Network Observability CLI filters and visualizes network flow and packet telemetry directly within the terminal. The tool exports captured data as JSON, database files, or Packet Capture (PCAP) files for seamless integration with third-party analysis utilities.
18.2.1. Capture flows Copy linkLink copied to clipboard!
Capture network flows and apply filters based on resources or zones directly in the CLI. This helps you solve complex use cases, such as visualizing the Round-Trip Time (RTT) between two different zones.
Table visualization in the CLI provides viewing and flow search capabilities.
Prerequisites
-
Install the OpenShift CLI (
oc). -
Install the Network Observability CLI (
oc netobserv) plugin.
Procedure
Capture flows with filters enabled by running the following command:
$ oc netobserv flows --enable_filter=true --action=Accept --cidr=0.0.0.0/0 --protocol=TCP --port=49051Add filters to the
live table filterprompt in the terminal to further refine the incoming flows. For example:live table filter: [SrcK8S_Zone:us-west-1b] press enter to match multiple regular expressions at once- Use the PageUp and PageDown keys to toggle between None, Resource, Zone, Host, Owner and all of the above.
-
To stop capturing, press Ctrl+C. The data that was captured is written to two separate files in an
./outputdirectory located in the same path used to install the CLI. View the captured data in the
./output/flow/<capture_date_time>.jsonJSON file, which contains JSON arrays of the captured data.Example JSON file
{ "AgentIP": "10.0.1.76", "Bytes": 561, "DnsErrno": 0, "Dscp": 20, "DstAddr": "f904:ece9:ba63:6ac7:8018:1e5:7130:0", "DstMac": "0A:58:0A:80:00:37", "DstPort": 9999, "Duplicate": false, "Etype": 2048, "Flags": 16, "FlowDirection": 0, "IfDirection": 0, "Interface": "ens5", "K8S_FlowLayer": "infra", "Packets": 1, "Proto": 6, "SrcAddr": "3e06:6c10:6440:2:a80:37:b756:270f", "SrcMac": "0A:58:0A:80:00:01", "SrcPort": 46934, "TimeFlowEndMs": 1709741962111, "TimeFlowRttNs": 121000, "TimeFlowStartMs": 1709741962111, "TimeReceived": 1709741964 }You can use SQLite to inspect the
./output/flow/<capture_date_time>.dbdatabase file. For example:Open the file by running the following command:
$ sqlite3 ./output/flow/<capture_date_time>.dbQuery the data by running a SQLite
SELECTstatement, for example:sqlite> SELECT DnsLatencyMs, DnsFlagsResponseCode, DnsId, DstAddr, DstPort, Interface, Proto, SrcAddr, SrcPort, Bytes, Packets FROM flow WHERE DnsLatencyMs >10 LIMIT 10;Example output
12|NoError|58747|10.128.0.63|57856||17|172.30.0.10|53|284|1 11|NoError|20486|10.128.0.52|56575||17|169.254.169.254|53|225|1 11|NoError|59544|10.128.0.103|51089||17|172.30.0.10|53|307|1 13|NoError|32519|10.128.0.52|55241||17|169.254.169.254|53|254|1 12|NoError|32519|10.0.0.3|55241||17|169.254.169.254|53|254|1 15|NoError|57673|10.128.0.19|59051||17|172.30.0.10|53|313|1 13|NoError|35652|10.0.0.3|46532||17|169.254.169.254|53|183|1 32|NoError|37326|10.0.0.3|52718||17|169.254.169.254|53|169|1 14|NoError|14530|10.0.0.3|58203||17|169.254.169.254|53|246|1 15|NoError|40548|10.0.0.3|45933||17|169.254.169.254|53|174|1
18.2.2. Capture packets Copy linkLink copied to clipboard!
Use the Network Observability CLI to capture network packets. You can apply filters and refine them live in the terminal for accurate, real-time debugging.
Prerequisites
-
Install the OpenShift CLI (
oc). -
Install the Network Observability CLI (
oc netobserv) plugin.
Procedure
Run the packet capture with filters enabled:
$ oc netobserv packets --action=Accept --cidr=0.0.0.0/0 --protocol=TCP --port=49051Add filters to the
live table filterprompt in the terminal to refine the incoming packets. An example filter is as follows:live table filter: [SrcK8S_Zone:us-west-1b] press enter to match multiple regular expressions at once- Use the PageUp and PageDown keys to toggle between None, Resource, Zone, Host, Owner and all of the above.
- To stop capturing, press Ctrl+C.
View the captured data, which is written to a single file in an
./output/pcapdirectory located in the same path that was used to install the CLI:-
The
./output/pcap/<capture_date_time>.pcapfile can be opened with Wireshark.
-
The
18.2.3. Capture metrics Copy linkLink copied to clipboard!
Generate on-demand network observability dashboards in Prometheus using a service monitor. This allows you to quickly view and analyze network metrics.
Prerequisites
-
Install the OpenShift CLI (
oc). -
Install the Network Observability CLI (
oc netobserv) plugin.
Procedure
Capture metrics with filters enabled by running the following command:
Example output
$ oc netobserv metrics --enable_filter=true --cidr=0.0.0.0/0 --protocol=TCP --port=49051Open the link provided in the terminal to view the NetObserv / On-Demand dashboard:
Example URL
https://console-openshift-console.apps.rosa...openshiftapps.com/monitoring/dashboards/netobserv-cliNoteFeatures that are not enabled present as empty graphs.
18.2.4. Clean the Network Observability CLI Copy linkLink copied to clipboard!
Use oc netobserv cleanup to manually remove all components installed by the Network Observability CLI from your cluster. While the client runs this command automatically after a capture, you may need to run it manually if you face connectivity issues.
Procedure
Run the following command:
$ oc netobserv cleanup
Additional resources
18.3. Capture plain text from OpenSSL encrypted network traffic Copy linkLink copied to clipboard!
Use the Network Observability CLI to capture and view plain text content from OpenSSL-encrypted network packets. This CLI-only feature uses eBPF probes to intercept and display unencrypted data without requiring TLS certificates or Operator configuration.
18.3.1. Plain text packet capture for OpenSSL encrypted traffic Copy linkLink copied to clipboard!
The Network Observability CLI packet capture feature can intercept and display the unencrypted content of OpenSSL-encrypted network traffic.
This feature helps you debug encrypted connections when you need to inspect payload data that would otherwise be unreadable.
The CLI uses eBPF probes to intercept OpenSSL library functions before encryption and after decryption, capturing the plain text content without requiring TLS certificates or keys.
This is a CLI-only feature that requires no Operator or agent configuration.
This capability is particularly valuable for the following tasks:
- Debugging application-level protocol issues in OpenSSL-encrypted traffic.
- Verifying API request and response payloads between services.
- Analyzing encrypted traffic content without disabling TLS.
- Troubleshooting encrypted connections where you lack access to certificates.
Plain text capture currently supports OpenSSL encrypted traffic only. The feature works with the oc netobserv packets subcommand for packet-level capture, not with flow-level capture.
18.3.1.1. How plain text capture works Copy linkLink copied to clipboard!
When you enable plain text capture during packet capture, the following actions occur:
- The CLI attaches eBPF probes to OpenSSL library functions in the target process.
- The probes intercept data before encryption (for outgoing traffic) and after decryption (for incoming traffic).
- Plain text content is captured and displayed alongside standard packet information.
- You can filter captures by pod IP, port, or other criteria to focus on specific traffic.
18.3.1.2. Key considerations Copy linkLink copied to clipboard!
When using plain text capture, consider the following conditions:
- Privileged access
- The CLI requires sufficient permissions to attach eBPF probes to running processes. You must have cluster-admin privileges or equivalent access.
- OpenSSL only
-
The feature captures plain text from applications that use dynamically linked
OpenSSLorlibsslonly. Applications that use staticOpenSSLor other TLS libraries are not supported. - Packet capture only
-
Plain text capture works with
oc netobserv packetsfor packet-level data, not withoc netobserv flowsfor flow-level aggregation. - Performance impact
- eBPF probes have minimal performance impact, but capturing large volumes of plain text data can consume additional resources. Use filters to limit capture scope in production environments.
- Security and compliance
- Captured plain text data can contain sensitive information such as passwords, API keys, or personal data. Ensure appropriate access controls and retention policies are in place.
18.3.2. Prepare for plain text packet capture Copy linkLink copied to clipboard!
Verify that your environment meets the requirements for capturing plain text from OpenSSL-encrypted network traffic.
Prerequisites
-
The OpenShift CLI (
oc) is installed. -
The Network Observability CLI (
oc netobserv) plugin is installed. -
You have
cluster-adminprivileges or equivalent access to run privileged operations.
Procedure
Verify the Network Observability CLI plugin is installed by running the following command:
$ oc netobserv versionExample output
NetObserv CLI version 1.12.0Identify the target application for packet capture by running the following command:
$ oc get pods -n <namespace>-
Replace
<namespace>with the name of the namespace where your pods are deployed. - Note the pod name and namespace of the application that uses OpenSSL for TLS encryption.
-
Replace
Get the pod IP address by running the following command:
$ oc get pod <pod_name> -n <namespace> -o jsonpath='{.status.podIP}'Identify the port the application listens on by running the following command:
$ oc get pod <pod_name> -n <namespace> -o jsonpath='{.spec.containers[*].ports[*].containerPort}'
Verification
Verify that you have the information needed for packet capture by using the following examples:
-
Pod IP address, such as
10.131.0.15 -
Port number, such as
8443 - Pod uses OpenSSL for TLS encryption
-
Pod IP address, such as
You can now proceed to capture packets with OpenSSL plain text.
18.3.3. Capture packets that contain OpenSSL plain text Copy linkLink copied to clipboard!
Capture network packets and view the unencrypted plain text content from OpenSSL-encrypted connections by using the Network Observability CLI.
Prerequisites
-
The OpenShift CLI (
oc) is installed. -
The Network Observability CLI (
oc netobserv) plugin is installed. -
You have
cluster-adminprivileges. - You have identified the pod IP address and port of the target application that uses OpenSSL.
Plain text packet capture reveals unencrypted content from TLS connections, which can contain sensitive information such as passwords, API keys, or personal data. Ensure that appropriate access controls and retention policies are in place. Delete capture files securely when no longer needed.
Procedure
Start a packet capture with OpenSSL plain text enabled, filtering by the target pod IP and port:
$ oc netobserv packets --peer_ip=<pod_ip> --port=<pod_port> --enable_opensslReplace
<pod_ip>with the pod IP address and<pod_port>with the port number.Example
$ oc netobserv packets --peer_ip=10.131.0.15 --port=8443 --enable_opensslThe CLI displays a live table of captured packets with plain text data from OpenSSL-encrypted connections.
- Pause the capture by pressing Ctrl+Space.
Navigate the packet capture output:
- Use the arrow keys (↑ ↓) to select a packet.
- When the capture is paused, the selected packet’s payload is displayed in the detail panel.
- Press Tab to cycle focus between the filter input, packet table, and detail panel.
- Press Esc to resume the capture.
- You can also use the mouse to select packets and move focus between panels.
Optional: Apply additional filters to narrow the capture scope:
$ oc netobserv packets --peer_ip=<pod_ip> --port=<pod_port> --query='SrcK8S_Namespace="<namespace>" or DstK8S_Namespace="<namespace>"' --protocol=TCP --enable_opensslYou can combine filters such as
--protocol,--query, or other options to focus on specific traffic. Use--queryto filter by namespace, as--namespacechanges where the CLI components are deployed.Optional: Capture packets to a file for later analysis:
$ oc netobserv packets --peer_ip=<pod_ip> --port=<pod_port> --enable_openssl --copyThe CLI writes captured packet data, including plain text content, to the
./output/pcapdirectory.Stop the packet capture:
Press Ctrl+C to stop capturing packets and return to the shell prompt.
Verification
If you captured packets to a file, verify the output:
$ ls ./output/pcap/The directory contains timestamped
.pcapngcapture files. Open a capture file with Wireshark for packet-level analysis.Verify the captured plain text:
$ ls ./output/plaintext/The directory contains timestamped
.jsonlfiles with decoded plain text records.Examine a plain text capture:
$ less ./output/plaintext/<timestamp>.jsonl
18.4. Network Observability CLI (oc netobserv) reference Copy linkLink copied to clipboard!
The Network Observability CLI (oc netobserv) provides feature and filtering parity with the Network Observability Operator. Use command-line arguments to dynamically toggle features and isolate specific cluster network traffic flows.
18.4.1. Network Observability CLI usage Copy linkLink copied to clipboard!
You can use the Network Observability CLI (oc netobserv) to pass command line arguments to capture flows data, packets data, and metrics for further analysis and enable features supported by the Network Observability Operator.
18.4.1.1. Syntax Copy linkLink copied to clipboard!
The basic syntax for oc netobserv commands:
oc netobserv syntax
$ oc netobserv [<command>] [<feature_option>] [<command_options>]
- 1
- Feature options can only be used with the
oc netobserv flowscommand. They cannot be used with theoc netobserv packetscommand.
18.4.1.2. Basic commands Copy linkLink copied to clipboard!
| Command | Description |
|---|---|
| flows | Capture flows information. For subcommands, see the "Flows capture options" table. |
| packets | Capture packets data. For subcommands, see the "Packets capture options" table. |
| metrics | Capture metrics data. For subcommands, see the "Metrics capture options" table. |
| follow | Follow collector logs when running in background. |
| stop | Stop collection by removing agent daemonset. |
| copy | Copy collector generated files locally. |
| cleanup | Remove the Network Observability CLI components. |
| version | Print the software version. |
| help | Show help. |
18.4.1.3. Flows capture options Copy linkLink copied to clipboard!
Flows capture has mandatory commands as well as additional options, such as enabling extra features about packet drops, DNS latencies, Round-trip time, and filtering.
oc netobserv flows syntax
$ oc netobserv flows [<feature_option>] [<command_options>]
| Option | Description | Default |
|---|---|---|
| --enable_all | enable all eBPF features | false |
| --enable_dns | enable DNS tracking | false |
| --enable_ipsec | enable IPsec tracking | false |
| --enable_network_events | enable network events monitoring | false |
| --enable_pkt_translation | enable packet translation | false |
| --enable_pkt_drop | enable packet drop | false |
| --enable_rtt | enable RTT tracking | false |
| --enable_udn_mapping | enable User Defined Network mapping | false |
| --get-subnets | get subnets information | false |
| --privileged | force eBPF agent privileged mode | auto |
| --sampling | packets sampling interval | 1 |
| --background | run in background | false |
| --copy | copy the output files locally | prompt |
| --log-level | components logs | info |
| --max-time | maximum capture time | 5m |
| --max-bytes | maximum capture bytes | 50000000 = 50MB |
| --action | filter action | Accept |
| --cidr | filter CIDR | 0.0.0.0/0 |
| --direction | filter direction | - |
| --dport | filter destination port | - |
| --dport_range | filter destination port range | - |
| --dports | filter on either of two destination ports | - |
| --drops | filter flows with only dropped packets | false |
| --icmp_code | filter ICMP code | - |
| --icmp_type | filter ICMP type | - |
| --node-selector | capture on specific nodes | - |
| --peer_ip | filter peer IP | - |
| --peer_cidr | filter peer CIDR | - |
| --port_range | filter port range | - |
| --port | filter port | - |
| --ports | filter on either of two ports | - |
| --protocol | filter protocol | - |
| --query | filter flows using a custom query | - |
| --sport_range | filter source port range | - |
| --sport | filter source port | - |
| --sports | filter on either of two source ports | - |
| --tcp_flags | filter TCP flags | - |
| --interfaces | list of interfaces to monitor, comma separated | - |
| --exclude_interfaces | list of interfaces to exclude, comma separated | lo |
Example running flows capture on TCP protocol and port 49051 with PacketDrop and RTT features enabled:
$ oc netobserv flows --enable_pkt_drop --enable_rtt --action=Accept --cidr=0.0.0.0/0 --protocol=TCP --port=49051
18.4.1.4. Packets capture options Copy linkLink copied to clipboard!
You can filter packets capture data the as same as flows capture by using the filters. Certain features, such as packets drop, DNS, RTT, and network events, are only available for flows and metrics capture. To capture plaintext from dynamically linked OpenSSL applications, use --enable_openssl with a peer or port filter. The option requires a privileged agent and adds plaintext records to the packet capture output.
oc netobserv packets syntax
$ oc netobserv packets [<option>]
| Option | Description | Default |
|---|---|---|
| --background | run in background | false |
| --copy | copy the output files locally | prompt |
| --log-level | components logs | info |
| --max-time | maximum capture time | 5m |
| --max-bytes | maximum capture bytes | 50000000 = 50MB |
| --enable_openssl | capture TLS plaintext via OpenSSL uprobes | false; privileged agent; recommended: --peer_ip --port |
| --tls_process_allowlist | restrict TLS uprobes to comma-separated process names | all non-infra; e.g. nginx,envoy |
| --tls_plaintext_min_bytes | drop TLS plaintext events shorter than N bytes | 0; agent env TLS_PLAINTEXT_MIN_BYTES |
| --tls_plaintext_preview_bytes | PlaintextPreview length; zero means full payload | 256 |
| --tls-keylog | path to SSLKEYLOGFILE for pcapng decryption | collector setting |
| --action | filter action | Accept |
| --cidr | filter CIDR | 0.0.0.0/0 |
| --direction | filter direction | - |
| --dport | filter destination port | - |
| --dport_range | filter destination port range | - |
| --dports | filter on either of two destination ports | - |
| --drops | filter flows with only dropped packets | false |
| --icmp_code | filter ICMP code | - |
| --icmp_type | filter ICMP type | - |
| --node-selector | capture on specific nodes | - |
| --peer_ip | filter peer IP | - |
| --peer_cidr | filter peer CIDR | - |
| --port_range | filter port range | - |
| --port | filter port | - |
| --ports | filter on either of two ports | - |
| --protocol | filter protocol | - |
| --query | filter flows using a custom query | - |
| --sport_range | filter source port range | - |
| --sport | filter source port | - |
| --sports | filter on either of two source ports | - |
| --tcp_flags | filter TCP flags | - |
Example running packets capture on TCP protocol and port 49051:
$ oc netobserv packets --action=Accept --cidr=0.0.0.0/0 --protocol=TCP --port=49051
18.4.1.5. Metrics capture options Copy linkLink copied to clipboard!
You can enable features and use filters on metrics capture, the same as flows capture. The generated graphs fill accordingly in the dashboard.
oc netobserv metrics syntax
$ oc netobserv metrics [<option>]
| Option | Description | Default |
|---|---|---|
| --enable_all | enable all eBPF features | false |
| --enable_dns | enable DNS tracking | false |
| --enable_ipsec | enable IPsec tracking | false |
| --enable_network_events | enable network events monitoring | false |
| --enable_pkt_translation | enable packet translation | false |
| --enable_pkt_drop | enable packet drop | false |
| --enable_rtt | enable RTT tracking | false |
| --enable_udn_mapping | enable User Defined Network mapping | false |
| --get-subnets | get subnets information | false |
| --privileged | force eBPF agent privileged mode | auto |
| --sampling | packets sampling interval | 1 |
| --background | run in background | false |
| --log-level | components logs | info |
| --max-time | maximum capture time | 1h |
| --action | filter action | Accept |
| --cidr | filter CIDR | 0.0.0.0/0 |
| --direction | filter direction | - |
| --dport | filter destination port | - |
| --dport_range | filter destination port range | - |
| --dports | filter on either of two destination ports | - |
| --drops | filter flows with only dropped packets | false |
| --icmp_code | filter ICMP code | - |
| --icmp_type | filter ICMP type | - |
| --node-selector | capture on specific nodes | - |
| --peer_ip | filter peer IP | - |
| --peer_cidr | filter peer CIDR | - |
| --port_range | filter port range | - |
| --port | filter port | - |
| --ports | filter on either of two ports | - |
| --protocol | filter protocol | - |
| --query | filter flows using a custom query | - |
| --sport_range | filter source port range | - |
| --sport | filter source port | - |
| --sports | filter on either of two source ports | - |
| --tcp_flags | filter TCP flags | - |
| --include_list | list of metric names to generate, comma separated | namespace_flows_total,node_ingress_bytes_total,node_egress_bytes_total,workload_ingress_bytes_total |
| --interfaces | list of interfaces to monitor, comma separated | - |
| --exclude_interfaces | list of interfaces to exclude, comma separated | lo |
Example running metrics capture for TCP drops
$ oc netobserv metrics --enable_pkt_drop --protocol=TCP