Red Hat SummitChapter 12. Network Observability CLI
12.1. Installing the Network Observability CLI
The Network Observability CLI (oc netobserv) is deployed separately from the Network Observability Operator. The CLI is available as an OpenShift CLI (oc) plugin. It provides a lightweight way to quickly debug and troubleshoot with network observability.
12.1.1. About the Network Observability CLI
You can quickly debug and troubleshoot networking issues by using the Network Observability CLI (oc netobserv). The Network Observability CLI is a flow and packet visualization tool that relies on eBPF agents to stream collected data to an ephemeral collector pod. It requires no persistent storage during the capture. After the run, the output is transferred to your local machine. This enables quick, live insight into packets and flow data without installing the Network Observability Operator.
CLI capture is meant to run only for short durations, such as 8-10 minutes. If it runs for too long, it can be difficult to delete the running process.
12.1.2. Installing the Network Observability CLI
Installing the Network Observability CLI (oc netobserv) is a separate procedure from the Network Observability Operator installation. This means that, even if you have the Operator installed from OperatorHub, you need to install the CLI separately.
You can optionally use Krew to install the netobserv CLI plugin. For more information, see "Installing a CLI plugin with Krew".
Prerequisites
-
You must install the OpenShift CLI (
oc). - You must have a macOS or Linux operating system.
Procedure
-
Download the
oc netobservCLI tar file that corresponds with your architecture. Unpack the archive. For example, for the
amd64archive, run the following command:$ tar xvf netobserv-cli-linux-amd64.tar.gz
Make the file executable:
$ chmod +x ./oc-netobserv
Move the extracted
netobserv-clibinary to a directory that is on yourPATH, such as/usr/local/bin/:$ sudo mv ./oc-netobserv /usr/local/bin/
Verification
Verify that
oc netobservis available:$ oc netobserv version
Example output
Netobserv CLI version <version>
Additional resources
12.2. Using the Network Observability CLI
You can visualize and filter the flows and packets data directly in the terminal to see specific usage, such as identifying who is using a specific port. The Network Observability CLI collects flows as JSON and database files or packets as a PCAP file, which you can use with third-party tools.
12.2.1. Capturing flows
You can capture flows and filter on any resource or zone in the data to solve use cases, such as displaying Round-Trip Time (RTT) between two zones. Table visualization in the CLI provides viewing and flow search capabilities.
Prerequisites
-
Install the OpenShift CLI (
oc). -
Install the Network Observability CLI (
oc netobserv) plugin.
Procedure
Capture flows with filters enabled by running the following command:
$ oc netobserv flows --enable_filter=true --action=Accept --cidr=0.0.0.0/0 --protocol=TCP --port=49051
Add filters to the
live table filterprompt in the terminal to further refine the incoming flows. For example:live table filter: [SrcK8S_Zone:us-west-1b] press enter to match multiple regular expressions at once
- Use the PageUp and PageDown keys to toggle between None, Resource, Zone, Host, Owner and all of the above.
-
To stop capturing, press Ctrl+C. The data that was captured is written to two separate files in an
./outputdirectory located in the same path used to install the CLI. View the captured data in the
./output/flow/<capture_date_time>.jsonJSON file, which contains JSON arrays of the captured data.Example JSON file
{ "AgentIP": "10.0.1.76", "Bytes": 561, "DnsErrno": 0, "Dscp": 20, "DstAddr": "f904:ece9:ba63:6ac7:8018:1e5:7130:0", "DstMac": "0A:58:0A:80:00:37", "DstPort": 9999, "Duplicate": false, "Etype": 2048, "Flags": 16, "FlowDirection": 0, "IfDirection": 0, "Interface": "ens5", "K8S_FlowLayer": "infra", "Packets": 1, "Proto": 6, "SrcAddr": "3e06:6c10:6440:2:a80:37:b756:270f", "SrcMac": "0A:58:0A:80:00:01", "SrcPort": 46934, "TimeFlowEndMs": 1709741962111, "TimeFlowRttNs": 121000, "TimeFlowStartMs": 1709741962111, "TimeReceived": 1709741964 }You can use SQLite to inspect the
./output/flow/<capture_date_time>.dbdatabase file. For example:Open the file by running the following command:
$ sqlite3 ./output/flow/<capture_date_time>.db
Query the data by running a SQLite
SELECTstatement, for example:sqlite> SELECT DnsLatencyMs, DnsFlagsResponseCode, DnsId, DstAddr, DstPort, Interface, Proto, SrcAddr, SrcPort, Bytes, Packets FROM flow WHERE DnsLatencyMs >10 LIMIT 10;
Example output
12|NoError|58747|10.128.0.63|57856||17|172.30.0.10|53|284|1 11|NoError|20486|10.128.0.52|56575||17|169.254.169.254|53|225|1 11|NoError|59544|10.128.0.103|51089||17|172.30.0.10|53|307|1 13|NoError|32519|10.128.0.52|55241||17|169.254.169.254|53|254|1 12|NoError|32519|10.0.0.3|55241||17|169.254.169.254|53|254|1 15|NoError|57673|10.128.0.19|59051||17|172.30.0.10|53|313|1 13|NoError|35652|10.0.0.3|46532||17|169.254.169.254|53|183|1 32|NoError|37326|10.0.0.3|52718||17|169.254.169.254|53|169|1 14|NoError|14530|10.0.0.3|58203||17|169.254.169.254|53|246|1 15|NoError|40548|10.0.0.3|45933||17|169.254.169.254|53|174|1
12.2.2. Capturing packets
You can capture packets using the Network Observability CLI.
Prerequisites
-
Install the OpenShift CLI (
oc). -
Install the Network Observability CLI (
oc netobserv) plugin.
Procedure
Run the packet capture with filters enabled:
$ oc netobserv packets --action=Accept --cidr=0.0.0.0/0 --protocol=TCP --port=49051
Add filters to the
live table filterprompt in the terminal to refine the incoming packets. An example filter is as follows:live table filter: [SrcK8S_Zone:us-west-1b] press enter to match multiple regular expressions at once
- Use the PageUp and PageDown keys to toggle between None, Resource, Zone, Host, Owner and all of the above.
- To stop capturing, press Ctrl+C.
View the captured data, which is written to a single file in an
./output/pcapdirectory located in the same path that was used to install the CLI:-
The
./output/pcap/<capture_date_time>.pcapfile can be opened with Wireshark.
-
The
12.2.3. Capturing metrics
You can generate on-demand dashboards in Prometheus by using a service monitor for Network Observability.
Prerequisites
-
Install the OpenShift CLI (
oc). -
Install the Network Observability CLI (
oc netobserv) plugin.
Procedure
Capture metrics with filters enabled by running the following command:
Example output
$ oc netobserv metrics --enable_filter=true --cidr=0.0.0.0/0 --protocol=TCP --port=49051
Open the link provided in the terminal to view the NetObserv / On-Demand dashboard:
Example URL
https://console-openshift-console.apps.rosa...openshiftapps.com/monitoring/dashboards/netobserv-cli
NoteFeatures that are not enabled present as empty graphs.
12.2.4. Cleaning the Network Observability CLI
You can manually clean the CLI workload by running oc netobserv cleanup. This command removes all the CLI components from your cluster.
When you end a capture, this command is run automatically by the client. You might be required to manually run it if you experience connectivity issues.
Procedure
Run the following command:
$ oc netobserv cleanup
Additional resources
12.3. Network Observability CLI (oc netobserv) reference
The Network Observability CLI (oc netobserv) has most features and filtering options that are available for the Network Observability Operator. You can pass command line arguments to enable features or filtering options.
12.3.1. Network Observability CLI usage
You can use the Network Observability CLI (oc netobserv) to pass command line arguments to capture flows data, packets data, and metrics for further analysis and enable features supported by the Network Observability Operator.
12.3.1.1. Syntax
The basic syntax for oc netobserv commands:
oc netobserv syntax
$ oc netobserv [<command>] [<feature_option>] [<command_options>] 1
12.3.1.2. Basic commands
| Command | Description |
|---|---|
| flows | Capture flows information. For subcommands, see the "Flows capture options" table. |
| packets | Capture packets data. For subcommands, see the "Packets capture options" table. |
| metrics | Capture metrics data. For subcommands, see the "Metrics capture options" table. |
| follow | Follow collector logs when running in background. |
| stop | Stop collection by removing agent daemonset. |
| copy | Copy collector generated files locally. |
| cleanup | Remove the Network Observability CLI components. |
| version | Print the software version. |
| help | Show help. |
12.3.1.3. Flows capture options
Flows capture has mandatory commands as well as additional options, such as enabling extra features about packet drops, DNS latencies, Round-trip time, and filtering.
oc netobserv flows syntax
$ oc netobserv flows [<feature_option>] [<command_options>]
| Option | Description | Default |
|---|---|---|
| --enable_all | enable all eBPF features | false |
| --enable_dns | enable DNS tracking | false |
| --enable_network_events | enable network events monitoring | false |
| --enable_pkt_translation | enable packet translation | false |
| --enable_pkt_drop | enable packet drop | false |
| --enable_rtt | enable RTT tracking | false |
| --enable_udn_mapping | enable User Defined Network mapping | false |
| --get-subnets | get subnets information | false |
| --background | run in background | false |
| --copy | copy the output files locally | prompt |
| --log-level | components logs | info |
| --max-time | maximum capture time | 5m |
| --max-bytes | maximum capture bytes | 50000000 = 50MB |
| --action | filter action | Accept |
| --cidr | filter CIDR | 0.0.0.0/0 |
| --direction | filter direction | – |
| --dport | filter destination port | – |
| --dport_range | filter destination port range | – |
| --dports | filter on either of two destination ports | – |
| --drops | filter flows with only dropped packets | false |
| --icmp_code | filter ICMP code | – |
| --icmp_type | filter ICMP type | – |
| --node-selector | capture on specific nodes | – |
| --peer_ip | filter peer IP | – |
| --peer_cidr | filter peer CIDR | – |
| --port_range | filter port range | – |
| --port | filter port | – |
| --ports | filter on either of two ports | – |
| --protocol | filter protocol | – |
| --regexes | filter flows using regular expression | – |
| --sport_range | filter source port range | – |
| --sport | filter source port | – |
| --sports | filter on either of two source ports | – |
| --tcp_flags | filter TCP flags | – |
| --interfaces | interfaces to monitor | – |
Example running flows capture on TCP protocol and port 49051 with PacketDrop and RTT features enabled:
$ oc netobserv flows --enable_pkt_drop --enable_rtt --enable_filter --action=Accept --cidr=0.0.0.0/0 --protocol=TCP --port=49051
12.3.1.4. Packets capture options
You can filter packets capture data the as same as flows capture by using the filters. Certain features, such as packets drop, DNS, RTT, and network events, are only available for flows and metrics capture.
oc netobserv packets syntax
$ oc netobserv packets [<option>]
| Option | Description | Default |
|---|---|---|
| --background | run in background | false |
| --copy | copy the output files locally | prompt |
| --log-level | components logs | info |
| --max-time | maximum capture time | 5m |
| --max-bytes | maximum capture bytes | 50000000 = 50MB |
| --action | filter action | Accept |
| --cidr | filter CIDR | 0.0.0.0/0 |
| --direction | filter direction | – |
| --dport | filter destination port | – |
| --dport_range | filter destination port range | – |
| --dports | filter on either of two destination ports | – |
| --drops | filter flows with only dropped packets | false |
| --icmp_code | filter ICMP code | – |
| --icmp_type | filter ICMP type | – |
| --node-selector | capture on specific nodes | – |
| --peer_ip | filter peer IP | – |
| --peer_cidr | filter peer CIDR | – |
| --port_range | filter port range | – |
| --port | filter port | – |
| --ports | filter on either of two ports | – |
| --protocol | filter protocol | – |
| --regexes | filter flows using regular expression | – |
| --sport_range | filter source port range | – |
| --sport | filter source port | – |
| --sports | filter on either of two source ports | – |
| --tcp_flags | filter TCP flags | – |
Example running packets capture on TCP protocol and port 49051:
$ oc netobserv packets --action=Accept --cidr=0.0.0.0/0 --protocol=TCP --port=49051
12.3.1.5. Metrics capture options
You can enable features and use filters on metrics capture, the same as flows capture. The generated graphs fill accordingly in the dashboard.
oc netobserv metrics syntax
$ oc netobserv metrics [<option>]
| Option | Description | Default |
|---|---|---|
| --enable_all | enable all eBPF features | false |
| --enable_dns | enable DNS tracking | false |
| --enable_network_events | enable network events monitoring | false |
| --enable_pkt_translation | enable packet translation | false |
| --enable_pkt_drop | enable packet drop | false |
| --enable_rtt | enable RTT tracking | false |
| --enable_udn_mapping | enable User Defined Network mapping | false |
| --get-subnets | get subnets information | false |
| --action | filter action | Accept |
| --cidr | filter CIDR | 0.0.0.0/0 |
| --direction | filter direction | – |
| --dport | filter destination port | – |
| --dport_range | filter destination port range | – |
| --dports | filter on either of two destination ports | – |
| --drops | filter flows with only dropped packets | false |
| --icmp_code | filter ICMP code | – |
| --icmp_type | filter ICMP type | – |
| --node-selector | capture on specific nodes | – |
| --peer_ip | filter peer IP | – |
| --peer_cidr | filter peer CIDR | – |
| --port_range | filter port range | – |
| --port | filter port | – |
| --ports | filter on either of two ports | – |
| --protocol | filter protocol | – |
| --regexes | filter flows using regular expression | – |
| --sport_range | filter source port range | – |
| --sport | filter source port | – |
| --sports | filter on either of two source ports | – |
| --tcp_flags | filter TCP flags | – |
| --interfaces | interfaces to monitor | – |
Example running metrics capture for TCP drops
$ oc netobserv metrics --enable_pkt_drop --enable_filter --protocol=TCP
