Red Hat SummitChapter 2. Network Observability Operator release notes
The Network Observability Operator enables administrators to observe and analyze network traffic flows for OpenShift Container Platform clusters.
These release notes track the development of the Network Observability Operator in the OpenShift Container Platform.
For an overview of the Network Observability Operator, see About network observability.
2.1. Network Observability Operator 1.9.1
The following advisory is available for the Network Observability Operator 1.9.1:
2.1.1. Bug fixes
-
Before this update, network flows were not observed on OpenShift Container Platform 4.15 due to an incorrect attach mode setting. This stopped users from monitoring network flows correctly, especially with certain catalogs. With this release, the default attach mode for OpenShift Container Platform versions older than 4.16.0 is set to
tc, so flows are now observed on OpenShift Container Platform 4.15. (NETOBSERV-2333) - Before this update, if an IPFIX collector restarted, configuring an IPFIX exporter could lose its connection and stop sending network flows to the collector. With this release, the connection is restored, and network flows continue to be sent to the collector. (NETOBSERV-2315)
- Before this update, when you configured an IPFIX exporter, flows without port information (such as ICMP traffic) were ignored, which caused errors in logs. TCP flags and ICMP data were also missing from IPFIX exports. With this release, these details are now included. Missing fields (like ports) no longer cause errors and are part of the exported data. (NETOBSERV-2307)
- Before this update, the User Defined Networks (UDN) Mapping feature showed a configuration issue and warning on OpenShift Container Platform 4.18 because the OpenShift version was incorrectly set in the code. This impacted the user experience. With this release, UDN Mapping now supports OpenShift Container Platform 4.18 without warnings, making the user experience smooth. (NETOBSERV-2305)
-
Before this update, the expand function on the Network Traffic page had compatibility problems with OpenShift Container Platform Console 4.19. This resulted in empty menu space when expanding and an inconsistent user interface. With this release, the compatibility problem in the
NetflowTrafficpart andtheme hookis resolved. The side menu in the Network Traffic view is now properly managed, which improves how you interact with the user interface. (NETOBSERV-2304)
2.2. Network Observability Operator 1.9
The following advisory is available for the Network Observability Operator 1.9:
2.2.1. New features and enhancements
2.2.1.1. User-defined networks with network observability
With this release, user-defined networks (UDN) feature is generally available with network observability. When the UDNMapping feature is enabled in network observability, the Traffic flow table has a UDN labels column. You can filter logs on Source Network Name and Destination Network Name information.
2.2.1.2. Filter flowlogs at ingestion
With this release, you can create filters to reduce the number of generated network flows and the resource usage of network observability components. The following filters can be configured:
- eBPF Agent filters
- Flowlogs-pipeline filters
2.2.1.3. IPsec support
This update brings the following enhancements to network observability when IPsec is enabled on OpenShift Container Platform:
- A new column named IPsec Status is displayed in the network observability Traffic flows view to show whether a flow was successfully IPsec-encrypted or if there was an error during encryption/decryption.
- A new dashboard showing the percentage of encrypted traffic is generated.
2.2.1.4. Network Observability CLI
The following filtering options are now available for packets, flows, and metrics capture:
-
Configure the ratio of packets being sampled by using the
--samplingoption. -
Filter flows using a custom query by using the
--queryoption. -
Specify interfaces to monitor by using the
--interfacesoption. -
Specify interfaces to exclude by using the
--exclude_interfacesoption. -
Specify metric names to generate by using the
--include_listoption.
For more information, see Network Observability CLI reference.
2.2.2. Notable technical changes
-
The
NetworkEventsfeature in network observability 1.9 has been updated to work with the newer Linux kernel of OpenShift Container Platform 4.19. This update breaks compatibility with older kernels. As a result, theNetworkEventsfeature can only be used with OpenShift Container Platform 4.19. If you are using this feature with network observability 1.8 and OpenShift Container Platform 4.18, consider avoiding a network observability upgrade or upgrade to network observability 1.9 and OpenShift Container Platform to 4.19. -
The
netobserv-readercluster role has been renamed tonetobserv-loki-reader. - Improved CPU performance of the eBPF agents.
2.2.3. Technology Preview features
Some features in this release are currently in Technology Preview. These experimental features are not intended for production use. Note the following scope of support on the Red Hat Customer Portal for these features:
Technology Preview Features Support Scope
2.2.3.1. eBPF Manager Operator with network observability
The eBPF Manager Operator reduces the attack surface and ensures compliance, security, and conflict prevention by managing all eBPF programs. Network observability can use the eBPF Manager Operator to load hooks. This eliminates the need to provide the eBPF Agent with privileged mode or additional Linux capabilities like CAP_BPF and CAP_PERFMON. The eBPF Manager Operator with network observability is only supported on 64-bit AMD architecture.
2.2.4. CVE
2.2.5. Bug fixes
-
Previously, when filtering by source or destination IP from the console plugin, using a Classless Inter-Domain Routing (CIDR) notation such as
10.128.0.0/24did not work, returning results that should be filtered out. With this update, it is now possible to use a CIDR notation, with the results being filtered as expected. (NETOBSERV-2276) -
Previously, network flows might have incorrectly identified the network interfaces in use, especially with a risk of mixing up
eth0andens5. This issue only occurred when the eBPF agents were configured asPrivileged. With this update, it has been fixed partially, and almost all network interfaces are correctly identified. Refer to the known issues below for more details. (NETOBSERV-2257) - Previously, when the Operator checked for available Kubernetes APIs in order to adapt its behavior, if there was a stale API, this resulted in an error that prevented the Operator from starting normally. With this update, the Operator ignores error on unrelated APIs, logs errors on related APIs, and continues to run normally. (NETOBSERV-2240)
- Previously, users could not sort flows by Bytes or Packets in the Traffic flows view of the Console plugin. With this update, users can sort flows by Bytes and Packets. (NETOBSERV-2239)
-
Previously, when configuring the
FlowCollectorresource with an IPFIX exporter, MAC addresses in the IPFIX flows were truncated to their 2 first bytes. With this update, MAC addresses are fully represented in the IPFIX flows. (NETOBSERV-2208) - Previously, some of the warnings sent from the Operator validation webhook could lack clarity on what needed to be done. With this update, some of these messages have been reviewed and amended to make them more actionable. (NETOBSERV-2178)
-
Previously, it was not obvious to figure out there was an issue when referencing a
LokiStackfrom theFlowCollectorresource, such as in case of typing error. With this update, theFlowCollectorstatus clearly states that the referencedLokiStackis not found in that case. (NETOBSERV-2174) - Previously, in the console plugin Traffic flows view, in case of text overflow, text ellipses sometimes hid much of the text to be displayed. With this update, it displays as much text as possible. (NETOBSERV-2119)
- Previously, the console plugin for network observability 1.8.1 and earlier did not work with the OpenShift Container Platform 4.19 web console, making the Network Traffic page inaccessible. With this update, the console plugin is compatible and the Network Traffic page is accessible in network observability 1.9.0. (NETOBSERV-2046)
-
Previously, when using conversation tracking (
logTypes: ConversationsorlogTypes: Allin theFlowCollectorresource), the Traffic rates metrics visible in the dashboards were flawed, wrongly showing an out-of-control increase in traffic. Now, the metrics show more accurate traffic rates. However, note that inConversationsandEndedConversationsmodes, these metrics are still not completely accurate as they do not include long-standing connections. This information has been added to the documentation. The default modelogTypes: Flowsis recommended to avoid these inaccuracy. (NETOBSERV-1955)
2.2.6. Known issues
- The user-defined network (UDN) feature displays a configuration issue and a warning when used with OpenShift Container Platform 4.18, even though it is supported. This warning can be ignored. (NETOBSERV-2305)
-
In some rare cases, the eBPF agent is unable to appropriately correlate flows with the involved interfaces when running in
privilegedmodes with several network namespaces. A large part of these issues have been identified and resolved in this release, but some inconsistencies remain, especially with theens5interface. (NETOBSERV-2287)
2.3. Network Observability Operator 1.8.1
The following advisory is available for the Network Observability Operator 1.8.1:
2.3.1. CVEs
2.3.2. Bug fixes
- This fix ensures that the Observe menu appears only once in future versions of OpenShift Container Platform. (NETOBSERV-2139)
2.4. Network Observability Operator 1.8.0
The following advisory is available for the Network Observability Operator 1.8.0:
2.4.1. New features and enhancements
2.4.1.1. Packet translation
You can now enrich network flows with translated endpoint information, showing not only the service but also the specific backend pod, so you can see which pod served a request.
For more information, see Endpoint translation (xlat) and Working with endpoint translation (xlat).
2.4.1.2. OVN-Kubernetes networking events tracking
OVN-Kubernetes networking events tracking is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
You can now use network event tracking in network observability to gain insight into OVN-Kubernetes events, including network policies, admin network policies, and egress firewalls.
For more information, see Viewing network events.
2.4.1.3. eBPF performance improvements in 1.8
- Network observability now uses hash maps instead of per-CPU maps. This means that network flows data is now tracked in the kernel space and new packets are also aggregated there. The de-duplication of network flows can now occur in the kernel, so the size of data transfer between the kernel and the user spaces yields better performance. With these eBPF performance improvements, there is potential to observe a CPU resource reduction between 40% and 57% in the eBPF Agent.
2.4.1.4. Network Observability CLI
The following new features, options, and filters are added to the Network Observability CLI for this release:
-
Capture metrics with filters enabled by running the
oc netobserv metricscommand. -
Run the CLI in the background by using the
--backgroundoption with flows and packets capture and runningoc netobserv followto see the progress of the background run andoc netobserv copyto download the generated logs. -
Enrich flows and metrics capture with Machines, Pods, and Services subnets by using the
--get-subnetsoption. New filtering options available with packets, flows, and metrics capture:
- eBPF filters on IPs, Ports, Protocol, Action, TCP Flags and more
-
Custom nodes using
--node-selector -
Drops only using
--drops -
Any field using
--regexes
For more information, see Network Observability CLI reference.
2.4.2. Bug fixes
- Previously, the Network Observability Operator came with a "kube-rbac-proxy" container to manage RBAC for its metrics server. Since this external component is deprecated, it was necessary to remove it. It is now replaced with direct TLS and RBAC management through Kubernetes controller-runtime, without the need for a side-car proxy. (NETOBSERV-1999)
- Previously in the OpenShift Container Platform console plugin, filtering on a key that was not equal to multiple values would not filter anything. With this fix, the expected results are returned, which is all flows not having any of the filtered values. (NETOBSERV-1990)
- Previously in the OpenShift Container Platform console plugin with disabled Loki, it was very likely to generate a "Can’t build query" error due to selecting an incompatible set of filters and aggregations. Now this error is avoided avoid by automatically disabling incompatible filters while still making the user aware of the filter incompatibility. (NETOBSERV-1977)
- Previously, when viewing flow details from the console plugin, the ICMP info was always displayed in the side panel, showing "undefined" values for non-ICMP flows. With this fix, ICMP info is not displayed for non-ICMP flows. (NETOBSERV-1969)
- Previously, the "Export data" link from the Traffic flows view did not work as intended, generating empty CSV reports. Now, the export feature is restored, generating non-empty CSV data. (NETOBSERV-1958)
-
Previously, it was possible to configure the
FlowCollectorwithprocessor.logTypesConversations,EndedConversationsorAllwithloki.enableset tofalse, despite the conversation logs being only useful when Loki is enabled. This resulted in resource usage waste. Now, this configuration is invalid and is rejected by the validation webhook. (NETOBSERV-1957) -
Configuring the
FlowCollectorwithprocessor.logTypesset toAllconsumes much more resources, such as CPU, memory and network bandwidth, than the other options. This was previously not documented. It is now documented, and triggers a warning from the validation webhook. (NETOBSERV-1956) - Previously, under high stress, some flows generated by the eBPF agent were mistakenly dismissed, resulting in traffic bandwidth under-estimation. Now, those generated flows are not dismissed. (NETOBSERV-1954)
-
Previously, when enabling the network policy in the
FlowCollectorconfiguration, the traffic to the Operator webhooks was blocked, breaking theFlowMetricsAPI validation. Now traffic to the webhooks is allowed. (NETOBSERV-1934) -
Previously, when deploying the default network policy, namespaces
openshift-consoleandopenshift-monitoringwere set by default in theadditionalNamespacesfield, resulting in duplicated rules. Now there is no additional namespace set by default, which helps avoid getting duplicated rules.(NETOBSERV-1933) - Previously from the OpenShift Container Platform console plugin, filtering on TCP flags would match flows having only the exact desired flag. Now, any flow having at least the desired flag appears in filtered flows. (NETOBSERV-1890)
- When the eBPF agent runs in privileged mode and pods are continuously added or deleted, a file descriptor (FD) leak occurs. The fix ensures proper closure of the FD when a network namespace is deleted. (NETOBSERV-2063)
-
Previously, the CLI agent
DaemonSetdid not deploy on master nodes. Now, a toleration is added on the agentDaemonSetto schedule on every node when taints are set. Now, CLI agentDaemonSetpods run on all nodes. (NETOBSERV-2030) - Previously, the Source Resource and Source Destination filters autocomplete were not working when using Prometheus storage only. Now this issue is fixed and suggestions displays as expected. (NETOBSERV-1885)
- Previously, a resource using multiple IPs was displayed separately in the Topology view. Now, the resource shows as a single topology node in the view. (NETOBSERV-1818)
- Previously, the console refreshed the Network traffic table view contents when the mouse pointer hovered over the columns. Now, the display is fixed, so row height remains constant with a mouse hover. (NETOBSERV-2049)
2.4.3. Known issues
- If there is traffic that uses overlapping subnets in your cluster, there is a small risk that the eBPF Agent mixes up the flows from overlapped IPs. This can happen if different connections happen to have the exact same source and destination IPs and if ports and protocol are within a 5 seconds time frame and happening on the same node. This should not be possible unless you configured secondary networks or UDN. Even in that case, it is still very unlikely in usual traffic, as source ports are usually a good differentiator. (NETOBSERV-2115)
-
After selecting a type of exporter to configure in the
FlowCollectorresourcespec.exporterssection from the OpenShift Container Platform web console form view, the detailed configuration for that type does not show up in the form. The workaround is to configure directly the YAML. (NETOBSERV-1981)
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}