Chapter 6. BGP routing

Procedure

1. Enable no-overlay for the default network in the Cluster Network Operator (CNO) custom resource (CR).

   1. At installation time (day 0), set the BGP manifest in your installation manifests/ directory to configure the noOverlayConfig object such as in the following managed and unmanaged routing examples.

      Example CNO CR using no-overlay with unmanaged routing

      apiVersion: operator.openshift.io/v1
      kind: Network
      metadata:
        name: cluster
      spec:
        additionalRoutingCapabilities:
          providers:
          - FRR
        defaultNetwork:
          ovnKubernetesConfig:
            routeAdvertisements: Enabled
            transport: NoOverlay
            noOverlayConfig:
              outboundSNAT: Enabled
              routing: Unmanaged
          type: OVNKubernetes

      Example CNO CR using no-overlay with managed routing

      apiVersion: operator.openshift.io/v1
      kind: Network
      metadata:
        name: cluster
      spec:
        additionalRoutingCapabilities:
          providers:
          - FRR
        defaultNetwork:
          ovnKubernetesConfig:
            routeAdvertisements: Enabled
            transport: NoOverlay
            noOverlayConfig:
              outboundSNAT: Enabled
              routing: Managed
              bgpTopology: FullMesh
              asNumber: 64512
          type: OVNKubernetes

      spec.defaultNetwork.ovnKubernetesConfig.noOverlayConfig.bgpTopology

      Specifies FullMesh for a full-mesh BGP fabric between nodes.

      spec.defaultNetwork.ovnKubernetesConfig.noOverlayConfig.asNumber

      Optional: specifies the BGP autonomous system number used in the default VRF. When omitted, 64512 is used.

   2. On a running cluster (day 2), configure the noOverlayConfig object in CNO using the following command:

      $ oc patch network.operator.openshift.io cluster --type merge --patch '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"noOverlayConfig":{"outboundSNAT":"Enabled"}}}}}'

   3. For managed routing, proceed to the verification step. Do not create RouteAdvertisements or FRRConfiguration objects. OVN-Kubernetes can reconcile the managed BGP fabric

2. If you use unmanaged routing, add manifests to the installation manifests/ directory (day 0) for the following custom resources (CRs):

   1. Add the following FRRConfiguration CR:

      apiVersion: frrk8s.metallb.io/v1beta1
      kind: FRRConfiguration
      metadata:
        name: external-bgp
        namespace: openshift-frr-k8s
        labels:
          network: default
      spec:
        bgp:
          routers:
          - asn: 64512
            neighbors:
            - address: 192.168.111.1
              asn: 64512
              disableMP: true
              toReceive:
                allowed:
                  mode: filtered

      Replace spec.bgp.routers[].neighbors[].address, ASN values, and toReceive filters so they match your external BGP design.

   2. Add the following RouteAdvertisements CR:

      apiVersion: k8s.ovn.org/v1
      kind: RouteAdvertisements
      metadata:
        name: default
      spec:
        advertisements:
        - PodNetwork
        frrConfigurationSelector:
          matchLabels:
            network: default

3. Optional: Alternatively, you can create or apply the following CRs with oc apply (day 2):

   1. Create an FRRConfiguration CR that defines BGP peering toward your external router.

      Example FRRConfiguration CR for unmanaged routing

      apiVersion: frrk8s.metallb.io/v1beta1
      kind: FRRConfiguration
      metadata:
        name: external-bgp
        namespace: openshift-frr-k8s
        labels:
          network: default
      spec:
        bgp:
          routers:
          - asn: 64512
            neighbors:
            - address: 192.168.111.1
              asn: 64512
              disableMP: true
              toReceive:
                allowed:
                  mode: filtered

      Replace spec.bgp.routers[].neighbors[].address, ASN values, and toReceive filters so they match your external BGP design.

   2. Apply the FRRConfiguration CR using the following command:

      $ oc apply -f <frrconfiguration_file>.yaml

      Replace <frrconfiguration_file>.yaml with your manifest file name.

   3. Create a RouteAdvertisements CR that advertises the pod network.

      Example RouteAdvertisements CR for unmanaged routing

      apiVersion: k8s.ovn.org/v1
      kind: RouteAdvertisements
      metadata:
        name: default
      spec:
        advertisements:
        - PodNetwork
        frrConfigurationSelector:
          matchLabels:
            network: default
        networkSelectors:
        - networkSelectionType: DefaultNetwork
        nodeSelector: {}

      Note

      For unmanaged routing on the default network, at least one RouteAdvertisements object must select the default network. In the example, the spec.networkSelectors entry with networkSelectionType: DefaultNetwork selects the default network, spec.advertisements includes PodNetwork, and the RouteAdvertisements CR reaches Accepted=True in status. OVN-Kubernetes uses this configuration when advertising pod subnets to your BGP infrastructure.

   4. Apply the RouteAdvertisements CR using the following command:

      $ oc apply -f <routeadvertisements_file>.yaml

      Replace <routeadvertisements_file>.yaml with your file name.

Verification

1. Verify that the OVN-Kubernetes pods are running:

   $ oc get pods -n openshift-ovn-kubernetes

Procedure

1. Create a ClusterUserDefinedNetwork CR that uses no-overlay transport.

   Note

   For a primary layer 3 ClusterUserDefinedNetwork CR, every namespace that matches spec.namespaceSelector must include the k8s.ovn.org/primary-user-defined-network label before workloads can use the network; that label can only be set when the namespace is created.

   1. Set spec.network.noOverlayOptions.routing to Unmanaged.

      Example ClusterUserDefinedNetwork CR for no-overlay mode with outboundSNAT set to Disabled

      apiVersion: k8s.ovn.org/v1
      kind: ClusterUserDefinedNetwork
      metadata:
        name: high-perf-network
        labels:
          network: high-perf-network
      spec:
        namespaceSelector:
          matchLabels:
            app: performance-sensitive
        network:
          topology: Layer3
          layer3:
            role: Primary
            subnets:
            - cidr: 10.200.0.0/16
              hostSubnet: 24
          transport: "NoOverlay"
          noOverlayOptions:
            outboundSNAT: "Disabled"
            routing: "Unmanaged"

      Important

      Pods running on a CUDN running NoOverlay mode cannot establish TCP connections to NodePort services. This occurs when externalTrafficPolicy is set to Cluster and the backend pod resides on a different node than the one targeted by the request. This issue occurs regardless of whether outbound SNAT is enabled or disabled.

   2. Apply the ClusterUserDefinedNetwork CR by entering the following command:

      $ oc apply -f <cudn_file>.yaml

      Replace <cudn_file>.yaml with the name of your ClusterUserDefinedNetwork CR file.

2. Create a RouteAdvertisements CR

   1. Set spec.advertisements to PodNetwork to advertise the CUDN pod subnets to your external BGP infrastructure.

      Example RouteAdvertisements CR advertising the CUDN pod subnets

      apiVersion: k8s.ovn.org/v1
      kind: RouteAdvertisements
      metadata:
        name: high-perf-network
      spec:
        nodeSelector: {}
        frrConfigurationSelector:
          matchLabels:
            network: high-perf-network
        networkSelectors:
        - networkSelectionType: ClusterUserDefinedNetworks
          clusterUserDefinedNetworkSelector:
            networkSelector:
              matchLabels:
                network: high-perf-network
        advertisements:
        - PodNetwork

      where:

      spec.nodeSelector

      Specifies which nodes to include in the advertisements; when empty ({}), all nodes are selected.

      spec.frrConfigurationSelector

      Specifies the FRRConfiguration that peers with your external routers. Use matchLabels to select the FRRConfiguration by its labels.

      spec.networkSelectors.networkSelectionType

      Specifies the type of network to advertise. Set to ClusterUserDefinedNetworks to advertise a cluster user-defined network (CUDN). Set to DefaultNetwork to advertise the default cluster network.

      spec.advertisements

      Specifies the type of networks to advertise. Set to PodNetwork to advertise pod subnets. Set to EgressIP to advertise EgressIPs.

   2. Apply the RouteAdvertisements CR by entering the following command:

      $ oc apply -f <routeadvertisements_file>.yaml

      Replace <routeadvertisements_file>.yaml with the name of your RouteAdvertisements CR file.

3. Verify that the no-overlay transport was accepted by entering the following command:

   $ oc get clusteruserdefinednetwork high-perf-network -o yaml

Procedure

1. Verify that FRR-K8s pods are running by running the following command:

   $ oc get pods -n openshift-frr-k8s

2. If you configured no-overlay for the default network, verify the Cluster Network Operator (CNO) CR by running the following command:

   $ oc get network.operator cluster -o yaml

   Confirm that spec.defaultNetwork.ovnKubernetesConfig includes the expected routeAdvertisements, transport, and noOverlayConfig values.

3. If you use unmanaged routing, verify RouteAdvertisements objects by running the following commands:

   $ oc describe routeadvertisements <routeadvertisements_name>

   Replace <routeadvertisements_name> with the name of your RouteAdvertisements object.

   The output should show Accepted=True in the status section.

   Note

   If you use managed routing, you typically do not create RouteAdvertisements yourself for intra-cluster-only designs; if pod connectivity fails, continue with the remaining steps in this procedure.

4. If you configured a no-overlay ClusterUserDefinedNetwork CR, check its status by running the following command:

   $ oc get clusteruserdefinednetwork <cudn_name> -o yaml

   Replace <cudn_name> with the name of your CUDN CR.

5. Check for BGP-related errors in the OVN-Kubernetes logs by entering the following command:

   $ oc logs -n openshift-ovn-kubernetes -l app=ovnkube-node

6. Confirm that pod subnets are present in the node routing table by entering the following command:

   $ oc debug node/<node_name> -- chroot /host ip route

Procedure

1. To create the openshift-frr-k8s namespace, enter the following command:

   $ oc create namespace openshift-frr-k8s

2. To automate the migration, create a shell script named migrate.sh with the following contents:

   #!/bin/bash
   OLD_NAMESPACE="metallb-system"
   NEW_NAMESPACE="openshift-frr-k8s"
   FILTER_OUT="metallb-"
   oc get frrconfigurations.frrk8s.metallb.io -n "${OLD_NAMESPACE}" -o json |\
     jq -r '.items[] | select(.metadata.name | test("'"${FILTER_OUT}"'") | not)' |\
     jq -r '.metadata.namespace = "'"${NEW_NAMESPACE}"'"' |\
     oc create -f -

3. To execute the migration, run the following command:

   $ bash migrate.sh