Chapter 28. Load balancing with MetalLB

Procedure

1. Create a file, such as ipaddresspool.yaml, with content like the following example:

   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     namespace: metallb-system
     name: doc-example
     labels: 1
       zone: east
   spec:
     addresses:
     - 203.0.113.1-203.0.113.10
     - 203.0.113.65-203.0.113.75

   1

   This label assigned to the IPAddressPool can be referenced by the ipAddressPoolSelectors in the BGPAdvertisement CRD to associate the IPAddressPool with the advertisement.

2. Apply the configuration for the IP address pool:

   $ oc apply -f ipaddresspool.yaml

Procedure

1. Create a file, such as ipaddresspool-vlan.yaml, that is similar to the following example:

   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     namespace: metallb-system
     name: doc-example-vlan
     labels:
       zone: east 1
   spec:
     addresses:
     - 192.168.100.1-192.168.100.254 2

   1

   This label assigned to the IPAddressPool can be referenced by the ipAddressPoolSelectors in the BGPAdvertisement CRD to associate the IPAddressPool with the advertisement.

   2

   This IP range must match the subnet assigned to the VLAN on your network. To support layer 2 (L2) mode, the IP address range must be within the same subnet as the cluster nodes.

2. Apply the configuration for the IP address pool:

   $ oc apply -f ipaddresspool-vlan.yaml

3. To ensure this configuration applies to the VLAN you need to set the spec gatewayConfig.ipForwarding to Global.

   1. Run the following command to edit the network configuration custom resource (CR):

      $ oc edit network.config.openshift/cluster

   2. Update the spec.defaultNetwork.ovnKubernetesConfig section to include the gatewayConfig.ipForwarding set to Global. It should look something like this:

      Example

      ...
      spec:
        clusterNetwork:
          - cidr: 10.128.0.0/14
            hostPrefix: 23
        defaultNetwork:
          type: OVNKubernetes
          ovnKubernetesConfig:
            gatewayConfig:
              ipForwarding: Global
      ...

Procedure

1. Create an IP address pool by using a custom resource:

   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     namespace: metallb-system
     name: pool1
   spec:
     addresses:
       - 4.4.4.100-4.4.4.200
       - 2001:100:4::200-2001:100:4::400

2. Control which nodes in the cluster the IP address from pool1 advertises from by defining the .spec.nodeSelector value in the BGPAdvertisement custom resource:

   apiVersion: metallb.io/v1beta1
   kind: BGPAdvertisement
   metadata:
     name: example
   spec:
     ipAddressPools:
     - pool1
     nodeSelector:
     - matchLabels:
         kubernetes.io/hostname: NodeA
     - matchLabels:
         kubernetes.io/hostname: NodeB

Procedure

1. Create an IP address pool.

   1. Create a file, such as ipaddresspool.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: IPAddressPool
      metadata:
        namespace: metallb-system
        name: doc-example-l2
      spec:
        addresses:
          - 4.4.4.0/24
        autoAssign: false

   2. Apply the configuration for the IP address pool:

      $ oc apply -f ipaddresspool.yaml

2. Create a L2 advertisement.

   1. Create a file, such as l2advertisement.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: L2Advertisement
      metadata:
        name: l2advertisement
        namespace: metallb-system
      spec:
        ipAddressPools:
         - doc-example-l2

   2. Apply the configuration:

      $ oc apply -f l2advertisement.yaml

Procedure

1. Create an IP address pool.

   1. Create a file, such as ipaddresspool.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: IPAddressPool
      metadata:
        namespace: metallb-system
        name: doc-example-l2-label
        labels:
          zone: east
      spec:
        addresses:
          - 172.31.249.87/32

   2. Apply the configuration for the IP address pool:

      $ oc apply -f ipaddresspool.yaml

2. Create a L2 advertisement advertising the IP using ipAddressPoolSelectors.

   1. Create a file, such as l2advertisement.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: L2Advertisement
      metadata:
        name: l2advertisement-label
        namespace: metallb-system
      spec:
        ipAddressPoolSelectors:
          - matchExpressions:
              - key: zone
                operator: In
                values:
                  - east

   2. Apply the configuration:

      $ oc apply -f l2advertisement.yaml

Procedure

1. Create an IP address pool.

   1. Create a file, such as ipaddresspool.yaml, and enter the configuration details like the following example:

      apiVersion: metallb.io/v1beta1
      kind: IPAddressPool
      metadata:
        namespace: metallb-system
        name: doc-example-l2
      spec:
        addresses:
          - 4.4.4.0/24
        autoAssign: false

   2. Apply the configuration for the IP address pool like the following example:

      $ oc apply -f ipaddresspool.yaml

2. Create a L2 advertisement advertising the IP with interfaces selector.

   1. Create a YAML file, such as l2advertisement.yaml, and enter the configuration details like the following example:

      apiVersion: metallb.io/v1beta1
      kind: L2Advertisement
      metadata:
        name: l2advertisement
        namespace: metallb-system
      spec:
        ipAddressPools:
         - doc-example-l2
         interfaces:
         - interfaceA
         - interfaceB

   2. Apply the configuration for the advertisement like the following example:

      $ oc apply -f l2advertisement.yaml

Procedure

1. Enable forwarding for a specific secondary interface, such as bridge-net by creating and applying a MachineConfig CR.

   1. Create the MachineConfig CR to enable IP forwarding for the specified secondary interface named bridge-net.

   2. Save the following YAML in the enable-ip-forward.yaml file:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        labels:
          machineconfiguration.openshift.io/role: <node_role> 1
        name: 81-enable-global-forwarding
      spec:
        config:
          ignition:
            version: 3.2.0
          storage:
            files:
            - contents:
                source: data:text/plain;charset=utf-8;base64,`echo -e "net.ipv4.conf.bridge-net.forwarding = 1\nnet.ipv6.conf.bridge-net.forwarding = 1\nnet.ipv4.conf.bridge-net.rp_filter = 0\nnet.ipv6.conf.bridge-net.rp_filter = 0" | base64 -w0`
                 verification: {}
              filesystem: root
              mode: 644
              path: /etc/sysctl.d/enable-global-forwarding.conf
        osImageURL: ""

      1

      Node role where you want to enable IP forwarding, for example, worker

   3. Apply the configuration by running the following command:

      $ oc apply -f enable-ip-forward.yaml

2. Alternatively, you can enable IP forwarding globally by running the following command:

   $ oc patch network.operator cluster -p '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig":{"ipForwarding": "Global"}}}}}

Procedure

1. Create a file, such as bgppeer.yaml, with content like the following example:

   apiVersion: metallb.io/v1beta2
   kind: BGPPeer
   metadata:
     namespace: metallb-system
     name: doc-example-peer
   spec:
     peerAddress: 10.0.0.1
     peerASN: 64501
     myASN: 64500
     routerID: 10.10.10.10

2. Apply the configuration for the BGP peer:

   $ oc apply -f bgppeer.yaml

Procedure

1. Create address pool pool1.

   1. Create a file, such as ipaddresspool1.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: IPAddressPool
      metadata:
        namespace: metallb-system
        name: pool1
      spec:
        addresses:
          - 4.4.4.100-4.4.4.200
          - 2001:100:4::200-2001:100:4::400

   2. Apply the configuration for the IP address pool pool1:

      $ oc apply -f ipaddresspool1.yaml

2. Create address pool pool2.

   1. Create a file, such as ipaddresspool2.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: IPAddressPool
      metadata:
        namespace: metallb-system
        name: pool2
      spec:
        addresses:
          - 5.5.5.100-5.5.5.200
          - 2001:100:5::200-2001:100:5::400

   2. Apply the configuration for the IP address pool pool2:

      $ oc apply -f ipaddresspool2.yaml

3. Create BGP peer1.

   1. Create a file, such as bgppeer1.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta2
      kind: BGPPeer
      metadata:
        namespace: metallb-system
        name: peer1
      spec:
        peerAddress: 10.0.0.1
        peerASN: 64501
        myASN: 64500
        routerID: 10.10.10.10

   2. Apply the configuration for the BGP peer:

      $ oc apply -f bgppeer1.yaml

4. Create BGP peer2.

   1. Create a file, such as bgppeer2.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta2
      kind: BGPPeer
      metadata:
        namespace: metallb-system
        name: peer2
      spec:
        peerAddress: 10.0.0.2
        peerASN: 64501
        myASN: 64500
        routerID: 10.10.10.10

   2. Apply the configuration for the BGP peer2:

      $ oc apply -f bgppeer2.yaml

5. Create BGP advertisement 1.

   1. Create a file, such as bgpadvertisement1.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: BGPAdvertisement
      metadata:
        name: bgpadvertisement-1
        namespace: metallb-system
      spec:
        ipAddressPools:
          - pool1
        peers:
          - peer1
        communities:
          - 65535:65282
        aggregationLength: 32
        aggregationLengthV6: 128
        localPref: 100

   2. Apply the configuration:

      $ oc apply -f bgpadvertisement1.yaml

6. Create BGP advertisement 2.

   1. Create a file, such as bgpadvertisement2.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: BGPAdvertisement
      metadata:
        name: bgpadvertisement-2
        namespace: metallb-system
      spec:
        ipAddressPools:
          - pool2
        peers:
          - peer2
        communities:
          - 65535:65282
        aggregationLength: 32
        aggregationLengthV6: 128
        localPref: 100

   2. Apply the configuration:

      $ oc apply -f bgpadvertisement2.yaml

1. Define a BGP peer and add a network VRF instance.
2. Specify an IP address pool for MetalLB.
3. Configure a BGP route advertisement for MetalLB to advertise a route using the specified IP address pool and the BGP peer associated with the VRF instance.
4. Deploy a service to test the configuration.

Procedure

1. Create a BGPPeer custom resources (CR):

   1. Create a file, such as frrviavrf.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta2
      kind: BGPPeer
      metadata:
        name: frrviavrf
        namespace: metallb-system
      spec:
        myASN: 100
        peerASN: 200
        peerAddress: 192.168.130.1
        vrf: ens4vrf 1

      1

      Specifies the network VRF instance to associate with the BGP peer. MetalLB can advertise services and make routing decisions based on the routing information in the VRF.

      Note

      You must configure this network VRF instance in a NodeNetworkConfigurationPolicy CR. See the Additional resources for more information.

   2. Apply the configuration for the BGP peer by running the following command:

      $ oc apply -f frrviavrf.yaml

2. Create an IPAddressPool CR:

   1. Create a file, such as first-pool.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: IPAddressPool
      metadata:
        name: first-pool
        namespace: metallb-system
      spec:
        addresses:
        - 192.169.10.0/32

   2. Apply the configuration for the IP address pool by running the following command:

      $ oc apply -f first-pool.yaml

3. Create a BGPAdvertisement CR:

   1. Create a file, such as first-adv.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: BGPAdvertisement
      metadata:
        name: first-adv
        namespace: metallb-system
      spec:
        ipAddressPools:
          - first-pool
        peers:
          - frrviavrf 1

      1

      In this example, MetalLB advertises a range of IP addresses from the first-pool IP address pool to the frrviavrf BGP peer.

   2. Apply the configuration for the BGP advertisement by running the following command:

      $ oc apply -f first-adv.yaml

4. Create a Namespace, Deployment, and Service CR:

   1. Create a file, such as deploy-service.yaml, with content like the following example:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: test
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: server
        namespace: test
      spec:
        selector:
          matchLabels:
            app: server
        template:
          metadata:
            labels:
              app: server
          spec:
            containers:
            - name: server
              image: registry.redhat.io/ubi9/ubi
              ports:
              - name: http
                containerPort: 30100
              command: ["/bin/sh", "-c"]
              args: ["sleep INF"]
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: server1
        namespace: test
      spec:
        ports:
        - name: http
          port: 30100
          protocol: TCP
          targetPort: 30100
        selector:
          app: server
        type: LoadBalancer

   2. Apply the configuration for the namespace, deployment, and service by running the following command:

      $ oc apply -f deploy-service.yaml

Verification

1. Identify a MetalLB speaker pod by running the following command:

   $ oc get -n metallb-system pods -l component=speaker

   Example output

   NAME            READY   STATUS    RESTARTS   AGE
   speaker-c6c5f   6/6     Running   0          69m

2. Verify that the state of the BGP session is Established in the speaker pod by running the following command, replacing the variables to match your configuration:

   $ oc exec -n metallb-system <speaker_pod> -c frr -- vtysh -c "show bgp vrf <vrf_name> neigh"

   Example output

   BGP neighbor is 192.168.30.1, remote AS 200, local AS 100, external link
     BGP version 4, remote router ID 192.168.30.1, local router ID 192.168.30.71
     BGP state = Established, up for 04:20:09
   
   ...

3. Verify that the service is advertised correctly by running the following command:

   $ oc exec -n metallb-system <speaker_pod> -c frr -- vtysh -c "show bgp vrf <vrf_name> ipv4"

Procedure

1. Create an IP address pool.

   1. Create a file, such as ipaddresspool.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: IPAddressPool
      metadata:
        namespace: metallb-system
        name: doc-example-bgp-community
      spec:
        addresses:
          - 203.0.113.200/30
          - fc00:f853:ccd:e799::/124

   2. Apply the configuration for the IP address pool:

      $ oc apply -f ipaddresspool.yaml

2. Create a community alias named community1.

   apiVersion: metallb.io/v1beta1
   kind: Community
   metadata:
     name: community1
     namespace: metallb-system
   spec:
     communities:
       - name: NO_ADVERTISE
         value: '65535:65282'

3. Create a BGP peer named doc-example-bgp-peer.

   1. Create a file, such as bgppeer.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta2
      kind: BGPPeer
      metadata:
        namespace: metallb-system
        name: doc-example-bgp-peer
      spec:
        peerAddress: 10.0.0.1
        peerASN: 64501
        myASN: 64500
        routerID: 10.10.10.10

   2. Apply the configuration for the BGP peer:

      $ oc apply -f bgppeer.yaml

4. Create a BGP advertisement with the community alias.

   1. Create a file, such as bgpadvertisement.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: BGPAdvertisement
      metadata:
        name: bgp-community-sample
        namespace: metallb-system
      spec:
        aggregationLength: 32
        aggregationLengthV6: 128
        communities:
          - NO_ADVERTISE 1
        ipAddressPools:
          - doc-example-bgp-community
        peers:
          - doc-example-peer

      1

      Specify the CommunityAlias.name here and not the community custom resource (CR) name.

   2. Apply the configuration:

      $ oc apply -f bgpadvertisement.yaml

Procedure

1. Create a file, such as bfdprofile.yaml, with content like the following example:

   apiVersion: metallb.io/v1beta1
   kind: BFDProfile
   metadata:
     name: doc-example-bfd-profile-full
     namespace: metallb-system
   spec:
     receiveInterval: 300
     transmitInterval: 300
     detectMultiplier: 3
     echoMode: false
     passiveMode: true
     minimumTtl: 254

2. Apply the configuration for the BFD profile:

   $ oc apply -f bfdprofile.yaml

Procedure

1. Create a <service_name>.yaml file. In the file, ensure that the spec.type field is set to LoadBalancer.

   Refer to the examples for information about how to request the external IP address that MetalLB assigns to the service.

2. Create the service:

   $ oc apply -f <service_name>.yaml

   Example output

   service/<service_name> created

Procedure

1. Create a NodeNetworkConfigurationPolicy CR to define the VRF instance:

   1. Create a file, such as node-network-vrf.yaml, with content like the following example:

      apiVersion: nmstate.io/v1
      kind: NodeNetworkConfigurationPolicy
      metadata:
        name: vrfpolicy 1
      spec:
        nodeSelector:
          vrf: "true" 2
        maxUnavailable: 3
        desiredState:
          interfaces:
          - name: ens4vrf 3
            type: vrf 4
            state: up
            vrf:
              port:
              - ens4 5
              route-table-id: 2 6
          - name: ens4 7
            type: ethernet
            state: up
            ipv4:
              address:
              - ip: 192.168.130.130
                prefix-length: 24
              dhcp: false
              enabled: true
          routes: 8
            config:
            - destination: 0.0.0.0/0
              metric: 150
              next-hop-address: 192.168.130.1
              next-hop-interface: ens4
              table-id: 2
          route-rules: 9
            config:
            - ip-to: 172.30.0.0/16
              priority: 998
              route-table: 254 10
            - ip-to: 10.132.0.0/14
              priority: 998
              route-table: 254

      1

      The name of the policy.

      2

      This example applies the policy to all nodes with the label vrf:true.

      3

      The name of the interface.

      4

      The type of interface. This example creates a VRF instance.

      5

      The node interface that the VRF attaches to.

      6

      The name of the route table ID for the VRF.

      7

      The IPv4 address of the interface associated with the VRF.

      8

      Defines the configuration for network routes. The next-hop-address field defines the IP address of the next hop for the route. The next-hop-interface field defines the outgoing interface for the route. In this example, the VRF routing table is 2, which references the ID that you define in the EgressService CR.

      9

      Defines additional route rules. The ip-to fields must match the Cluster Network CIDR and Service Network CIDR. You can view the values for these CIDR address specifications by running the following command: oc describe network.config/cluster.

      10

      The main routing table that the Linux kernel uses when calculating routes has the ID 254.

   2. Apply the policy by running the following command:

      $ oc apply -f node-network-vrf.yaml

2. Create a BGPPeer custom resource (CR):

   1. Create a file, such as frr-via-vrf.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta2
      kind: BGPPeer
      metadata:
        name: frrviavrf
        namespace: metallb-system
      spec:
        myASN: 100
        peerASN: 200
        peerAddress: 192.168.130.1
        vrf: ens4vrf 1

      1

      Specifies the VRF instance to associate with the BGP peer. MetalLB can advertise services and make routing decisions based on the routing information in the VRF.

   2. Apply the configuration for the BGP peer by running the following command:

      $ oc apply -f frr-via-vrf.yaml

3. Create an IPAddressPool CR:

   1. Create a file, such as first-pool.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: IPAddressPool
      metadata:
        name: first-pool
        namespace: metallb-system
      spec:
        addresses:
        - 192.169.10.0/32

   2. Apply the configuration for the IP address pool by running the following command:

      $ oc apply -f first-pool.yaml

4. Create a BGPAdvertisement CR:

   1. Create a file, such as first-adv.yaml, with content like the following example:

      apiVersion: metallb.io/v1beta1
      kind: BGPAdvertisement
      metadata:
        name: first-adv
        namespace: metallb-system
      spec:
        ipAddressPools:
          - first-pool
        peers:
          - frrviavrf 1
        nodeSelectors:
          - matchLabels:
              egress-service.k8s.ovn.org/test-server1: "" 2

      1

      In this example, MetalLB advertises a range of IP addresses from the first-pool IP address pool to the frrviavrf BGP peer.

      2

      In this example, the EgressService CR configures the source IP address for egress traffic to use the load-balancer service IP address. Therefore, you must specify the load-balancer node for return traffic to use the same return path for the traffic originating from the pod.

   2. Apply the configuration for the BGP advertisement by running the following command:

      $ oc apply -f first-adv.yaml

5. Create an EgressService CR:

   1. Create a file, such as egress-service.yaml, with content like the following example:

      apiVersion: k8s.ovn.org/v1
      kind: EgressService
      metadata:
        name: server1 1
        namespace: test 2
      spec:
        sourceIPBy: "LoadBalancerIP" 3
        nodeSelector:
          matchLabels:
            vrf: "true" 4
        network: "2" 5

      1

      Specify the name for the egress service. The name of the EgressService resource must match the name of the load-balancer service that you want to modify.

      2

      Specify the namespace for the egress service. The namespace for the EgressService must match the namespace of the load-balancer service that you want to modify. The egress service is namespace-scoped.

      3

      This example assigns the LoadBalancer service ingress IP address as the source IP address for egress traffic.

      4

      If you specify LoadBalancer for the sourceIPBy specification, a single node handles the LoadBalancer service traffic. In this example, only a node with the label vrf: "true" can handle the service traffic. If you do not specify a node, OVN-Kubernetes selects a worker node to handle the service traffic. When a node is selected, OVN-Kubernetes labels the node in the following format: egress-service.k8s.ovn.org/<svc_namespace>-<svc_name>: "".

      5

      Specify the routing table for egress traffic.

   2. Apply the configuration for the egress service by running the following command:

      $ oc apply -f egress-service.yaml

Verification

1. Verify that you can access the application endpoint of the pods running behind the MetalLB service by running the following command:

   $ curl <external_ip_address>:<port_number> 1

   1

   Update the external IP address and port number to suit your application endpoint.

2. Optional: If you assigned the LoadBalancer service ingress IP address as the source IP address for egress traffic, verify this configuration by using tools such as tcpdump to analyze packets received at the external client.

Procedure

1. Create a file, such as setdebugloglevel.yaml, with content like the following example:

   apiVersion: metallb.io/v1beta1
   kind: MetalLB
   metadata:
     name: metallb
     namespace: metallb-system
   spec:
     logLevel: debug
     nodeSelector:
       node-role.kubernetes.io/worker: ""

2. Apply the configuration:

   $ oc replace -f setdebugloglevel.yaml

   Note

   Use oc replace as the understanding is the metallb CR is already created and here you are changing the log level.

3. Display the names of the speaker pods:

   $ oc get -n metallb-system pods -l component=speaker

   Example output

   NAME                    READY   STATUS    RESTARTS   AGE
   speaker-2m9pm           4/4     Running   0          9m19s
   speaker-7m4qw           3/4     Running   0          19s
   speaker-szlmx           4/4     Running   0          9m19s

   Note

   Speaker and controller pods are recreated to ensure the updated logging level is applied. The logging level is modified for all the components of MetalLB.

4. View the speaker logs:

   $ oc logs -n metallb-system speaker-7m4qw -c speaker

   Example output

   {"branch":"main","caller":"main.go:92","commit":"3d052535","goversion":"gc / go1.17.1 / amd64","level":"info","msg":"MetalLB speaker starting (commit 3d052535, branch main)","ts":"2022-05-17T09:55:05Z","version":""}
   {"caller":"announcer.go:110","event":"createARPResponder","interface":"ens4","level":"info","msg":"created ARP responder for interface","ts":"2022-05-17T09:55:05Z"}
   {"caller":"announcer.go:119","event":"createNDPResponder","interface":"ens4","level":"info","msg":"created NDP responder for interface","ts":"2022-05-17T09:55:05Z"}
   {"caller":"announcer.go:110","event":"createARPResponder","interface":"tun0","level":"info","msg":"created ARP responder for interface","ts":"2022-05-17T09:55:05Z"}
   {"caller":"announcer.go:119","event":"createNDPResponder","interface":"tun0","level":"info","msg":"created NDP responder for interface","ts":"2022-05-17T09:55:05Z"}
   I0517 09:55:06.515686      95 request.go:665] Waited for 1.026500832s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/operators.coreos.com/v1alpha1?timeout=32s
   {"Starting Manager":"(MISSING)","caller":"k8s.go:389","level":"info","ts":"2022-05-17T09:55:08Z"}
   {"caller":"speakerlist.go:310","level":"info","msg":"node event - forcing sync","node addr":"10.0.128.4","node event":"NodeJoin","node name":"ci-ln-qb8t3mb-72292-7s7rh-worker-a-vvznj","ts":"2022-05-17T09:55:08Z"}
   {"caller":"service_controller.go:113","controller":"ServiceReconciler","enqueueing":"openshift-kube-controller-manager-operator/metrics","epslice":"{\"metadata\":{\"name\":\"metrics-xtsxr\",\"generateName\":\"metrics-\",\"namespace\":\"openshift-kube-controller-manager-operator\",\"uid\":\"ac6766d7-8504-492c-9d1e-4ae8897990ad\",\"resourceVersion\":\"9041\",\"generation\":4,\"creationTimestamp\":\"2022-05-17T07:16:53Z\",\"labels\":{\"app\":\"kube-controller-manager-operator\",\"endpointslice.kubernetes.io/managed-by\":\"endpointslice-controller.k8s.io\",\"kubernetes.io/service-name\":\"metrics\"},\"annotations\":{\"endpoints.kubernetes.io/last-change-trigger-time\":\"2022-05-17T07:21:34Z\"},\"ownerReferences\":[{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"name\":\"metrics\",\"uid\":\"0518eed3-6152-42be-b566-0bd00a60faf8\",\"controller\":true,\"blockOwnerDeletion\":true}],\"managedFields\":[{\"manager\":\"kube-controller-manager\",\"operation\":\"Update\",\"apiVersion\":\"discovery.k8s.io/v1\",\"time\":\"2022-05-17T07:20:02Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:addressType\":{},\"f:endpoints\":{},\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:endpoints.kubernetes.io/last-change-trigger-time\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:app\":{},\"f:endpointslice.kubernetes.io/managed-by\":{},\"f:kubernetes.io/service-name\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"0518eed3-6152-42be-b566-0bd00a60faf8\\\"}\":{}}},\"f:ports\":{}}}]},\"addressType\":\"IPv4\",\"endpoints\":[{\"addresses\":[\"10.129.0.7\"],\"conditions\":{\"ready\":true,\"serving\":true,\"terminating\":false},\"targetRef\":{\"kind\":\"Pod\",\"namespace\":\"openshift-kube-controller-manager-operator\",\"name\":\"kube-controller-manager-operator-6b98b89ddd-8d4nf\",\"uid\":\"dd5139b8-e41c-4946-a31b-1a629314e844\",\"resourceVersion\":\"9038\"},\"nodeName\":\"ci-ln-qb8t3mb-72292-7s7rh-master-0\",\"zone\":\"us-central1-a\"}],\"ports\":[{\"name\":\"https\",\"protocol\":\"TCP\",\"port\":8443}]}","level":"debug","ts":"2022-05-17T09:55:08Z"}

5. View the FRR logs:

   $ oc logs -n metallb-system speaker-7m4qw -c frr

   Example output

   Started watchfrr
   2022/05/17 09:55:05 ZEBRA: client 16 says hello and bids fair to announce only bgp routes vrf=0
   2022/05/17 09:55:05 ZEBRA: client 31 says hello and bids fair to announce only vnc routes vrf=0
   2022/05/17 09:55:05 ZEBRA: client 38 says hello and bids fair to announce only static routes vrf=0
   2022/05/17 09:55:05 ZEBRA: client 43 says hello and bids fair to announce only bfd routes vrf=0
   2022/05/17 09:57:25.089 BGP: Creating Default VRF, AS 64500
   2022/05/17 09:57:25.090 BGP: dup addr detect enable max_moves 5 time 180 freeze disable freeze_time 0
   2022/05/17 09:57:25.090 BGP: bgp_get: Registering BGP instance (null) to zebra
   2022/05/17 09:57:25.090 BGP: Registering VRF 0
   2022/05/17 09:57:25.091 BGP: Rx Router Id update VRF 0 Id 10.131.0.1/32
   2022/05/17 09:57:25.091 BGP: RID change : vrf VRF default(0), RTR ID 10.131.0.1
   2022/05/17 09:57:25.091 BGP: Rx Intf add VRF 0 IF br0
   2022/05/17 09:57:25.091 BGP: Rx Intf add VRF 0 IF ens4
   2022/05/17 09:57:25.091 BGP: Rx Intf address add VRF 0 IF ens4 addr 10.0.128.4/32
   2022/05/17 09:57:25.091 BGP: Rx Intf address add VRF 0 IF ens4 addr fe80::c9d:84da:4d86:5618/64
   2022/05/17 09:57:25.091 BGP: Rx Intf add VRF 0 IF lo
   2022/05/17 09:57:25.091 BGP: Rx Intf add VRF 0 IF ovs-system
   2022/05/17 09:57:25.091 BGP: Rx Intf add VRF 0 IF tun0
   2022/05/17 09:57:25.091 BGP: Rx Intf address add VRF 0 IF tun0 addr 10.131.0.1/23
   2022/05/17 09:57:25.091 BGP: Rx Intf address add VRF 0 IF tun0 addr fe80::40f1:d1ff:feb6:5322/64
   2022/05/17 09:57:25.091 BGP: Rx Intf add VRF 0 IF veth2da49fed
   2022/05/17 09:57:25.091 BGP: Rx Intf address add VRF 0 IF veth2da49fed addr fe80::24bd:d1ff:fec1:d88/64
   2022/05/17 09:57:25.091 BGP: Rx Intf add VRF 0 IF veth2fa08c8c
   2022/05/17 09:57:25.091 BGP: Rx Intf address add VRF 0 IF veth2fa08c8c addr fe80::6870:ff:fe96:efc8/64
   2022/05/17 09:57:25.091 BGP: Rx Intf add VRF 0 IF veth41e356b7
   2022/05/17 09:57:25.091 BGP: Rx Intf address add VRF 0 IF veth41e356b7 addr fe80::48ff:37ff:fede:eb4b/64
   2022/05/17 09:57:25.092 BGP: Rx Intf add VRF 0 IF veth1295c6e2
   2022/05/17 09:57:25.092 BGP: Rx Intf address add VRF 0 IF veth1295c6e2 addr fe80::b827:a2ff:feed:637/64
   2022/05/17 09:57:25.092 BGP: Rx Intf add VRF 0 IF veth9733c6dc
   2022/05/17 09:57:25.092 BGP: Rx Intf address add VRF 0 IF veth9733c6dc addr fe80::3cf4:15ff:fe11:e541/64
   2022/05/17 09:57:25.092 BGP: Rx Intf add VRF 0 IF veth336680ea
   2022/05/17 09:57:25.092 BGP: Rx Intf address add VRF 0 IF veth336680ea addr fe80::94b1:8bff:fe7e:488c/64
   2022/05/17 09:57:25.092 BGP: Rx Intf add VRF 0 IF vetha0a907b7
   2022/05/17 09:57:25.092 BGP: Rx Intf address add VRF 0 IF vetha0a907b7 addr fe80::3855:a6ff:fe73:46c3/64
   2022/05/17 09:57:25.092 BGP: Rx Intf add VRF 0 IF vethf35a4398
   2022/05/17 09:57:25.092 BGP: Rx Intf address add VRF 0 IF vethf35a4398 addr fe80::40ef:2fff:fe57:4c4d/64
   2022/05/17 09:57:25.092 BGP: Rx Intf add VRF 0 IF vethf831b7f4
   2022/05/17 09:57:25.092 BGP: Rx Intf address add VRF 0 IF vethf831b7f4 addr fe80::f0d9:89ff:fe7c:1d32/64
   2022/05/17 09:57:25.092 BGP: Rx Intf add VRF 0 IF vxlan_sys_4789
   2022/05/17 09:57:25.092 BGP: Rx Intf address add VRF 0 IF vxlan_sys_4789 addr fe80::80c1:82ff:fe4b:f078/64
   2022/05/17 09:57:26.094 BGP: 10.0.0.1 [FSM] Timer (start timer expire).
   2022/05/17 09:57:26.094 BGP: 10.0.0.1 [FSM] BGP_Start (Idle->Connect), fd -1
   2022/05/17 09:57:26.094 BGP: Allocated bnc 10.0.0.1/32(0)(VRF default) peer 0x7f807f7631a0
   2022/05/17 09:57:26.094 BGP: sendmsg_zebra_rnh: sending cmd ZEBRA_NEXTHOP_REGISTER for 10.0.0.1/32 (vrf VRF default)
   2022/05/17 09:57:26.094 BGP: 10.0.0.1 [FSM] Waiting for NHT
   2022/05/17 09:57:26.094 BGP: bgp_fsm_change_status : vrf default(0), Status: Connect established_peers 0
   2022/05/17 09:57:26.094 BGP: 10.0.0.1 went from Idle to Connect
   2022/05/17 09:57:26.094 BGP: 10.0.0.1 [FSM] TCP_connection_open_failed (Connect->Active), fd -1
   2022/05/17 09:57:26.094 BGP: bgp_fsm_change_status : vrf default(0), Status: Active established_peers 0
   2022/05/17 09:57:26.094 BGP: 10.0.0.1 went from Connect to Active
   2022/05/17 09:57:26.094 ZEBRA: rnh_register msg from client bgp: hdr->length=8, type=nexthop vrf=0
   2022/05/17 09:57:26.094 ZEBRA: 0: Add RNH 10.0.0.1/32 type Nexthop
   2022/05/17 09:57:26.094 ZEBRA: 0:10.0.0.1/32: Evaluate RNH, type Nexthop (force)
   2022/05/17 09:57:26.094 ZEBRA: 0:10.0.0.1/32: NH has become unresolved
   2022/05/17 09:57:26.094 ZEBRA: 0: Client bgp registers for RNH 10.0.0.1/32 type Nexthop
   2022/05/17 09:57:26.094 BGP: VRF default(0): Rcvd NH update 10.0.0.1/32(0) - metric 0/0 #nhops 0/0 flags 0x6
   2022/05/17 09:57:26.094 BGP: NH update for 10.0.0.1/32(0)(VRF default) - flags 0x6 chgflags 0x0 - evaluate paths
   2022/05/17 09:57:26.094 BGP: evaluate_paths: Updating peer (10.0.0.1(VRF default)) status with NHT
   2022/05/17 09:57:30.081 ZEBRA: Event driven route-map update triggered
   2022/05/17 09:57:30.081 ZEBRA: Event handler for route-map: 10.0.0.1-out
   2022/05/17 09:57:30.081 ZEBRA: Event handler for route-map: 10.0.0.1-in
   2022/05/17 09:57:31.104 ZEBRA: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
   2022/05/17 09:57:31.104 ZEBRA: 	Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
   2022/05/17 09:57:31.105 ZEBRA: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
   2022/05/17 09:57:31.105 ZEBRA: 	Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring

Procedure

1. Display the names of the speaker pods:

   $ oc get -n metallb-system pods -l component=speaker

   Example output

   NAME            READY   STATUS    RESTARTS   AGE
   speaker-66bth   4/4     Running   0          56m
   speaker-gvfnf   4/4     Running   0          56m
   ...

2. Display the running configuration for FRR:

   $ oc exec -n metallb-system speaker-66bth -c frr -- vtysh -c "show running-config"

   Example output

   Building configuration...
   
   Current configuration:
   !
   frr version 7.5.1_git
   frr defaults traditional
   hostname some-hostname
   log file /etc/frr/frr.log informational
   log timestamp precision 3
   service integrated-vtysh-config
   !
   router bgp 64500  1
    bgp router-id 10.0.1.2
    no bgp ebgp-requires-policy
    no bgp default ipv4-unicast
    no bgp network import-check
    neighbor 10.0.2.3 remote-as 64500  2
    neighbor 10.0.2.3 bfd profile doc-example-bfd-profile-full  3
    neighbor 10.0.2.3 timers 5 15
    neighbor 10.0.2.4 remote-as 64500
    neighbor 10.0.2.4 bfd profile doc-example-bfd-profile-full
    neighbor 10.0.2.4 timers 5 15
    !
    address-family ipv4 unicast
     network 203.0.113.200/30   4
     neighbor 10.0.2.3 activate
     neighbor 10.0.2.3 route-map 10.0.2.3-in in
     neighbor 10.0.2.4 activate
     neighbor 10.0.2.4 route-map 10.0.2.4-in in
    exit-address-family
    !
    address-family ipv6 unicast
     network fc00:f853:ccd:e799::/124
     neighbor 10.0.2.3 activate
     neighbor 10.0.2.3 route-map 10.0.2.3-in in
     neighbor 10.0.2.4 activate
     neighbor 10.0.2.4 route-map 10.0.2.4-in in
    exit-address-family
   !
   route-map 10.0.2.3-in deny 20
   !
   route-map 10.0.2.4-in deny 20
   !
   ip nht resolve-via-default
   !
   ipv6 nht resolve-via-default
   !
   line vty
   !
   bfd
    profile doc-example-bfd-profile-full
     transmit-interval 35
     receive-interval 35
     passive-mode
     echo-mode
     echo-interval 35
     minimum-ttl 10
    !
   !
   end

   1

   The router bgp section indicates the ASN for MetalLB.

   2

   Confirm that a neighbor <ip-address> remote-as <peer-ASN> line exists for each BGP peer custom resource that you added.

   3

   If you configured BFD, confirm that the BFD profile is associated with the correct BGP peer and that the BFD profile appears in the command output.

   4

   Confirm that the network <ip-address-range> lines match the IP address ranges that you specified in address pool custom resources that you added.

3. Display the BGP summary:

   $ oc exec -n metallb-system speaker-66bth -c frr -- vtysh -c "show bgp summary"

   Example output

   IPv4 Unicast Summary:
   BGP router identifier 10.0.1.2, local AS number 64500 vrf-id 0
   BGP table version 1
   RIB entries 1, using 192 bytes of memory
   Peers 2, using 29 KiB of memory
   
   Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
   10.0.2.3        4      64500       387       389        0    0    0 00:32:02            0        1  1
   10.0.2.4        4      64500         0         0        0    0    0    never       Active        0  2
   
   Total number of neighbors 2
   
   IPv6 Unicast Summary:
   BGP router identifier 10.0.1.2, local AS number 64500 vrf-id 0
   BGP table version 1
   RIB entries 1, using 192 bytes of memory
   Peers 2, using 29 KiB of memory
   
   Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
   10.0.2.3        4      64500       387       389        0    0    0 00:32:02 NoNeg
   10.0.2.4        4      64500         0         0        0    0    0    never       Active        0
   
   Total number of neighbors 2

   1

   Confirm that the output includes a line for each BGP peer custom resource that you added.

   2

   Output that shows 0 messages received and messages sent indicates a BGP peer that does not have a BGP session. Check network connectivity and the BGP configuration of the BGP peer.

4. Display the BGP peers that received an address pool:

   $ oc exec -n metallb-system speaker-66bth -c frr -- vtysh -c "show bgp ipv4 unicast 203.0.113.200/30"

   Replace ipv4 with ipv6 to display the BGP peers that received an IPv6 address pool. Replace 203.0.113.200/30 with an IPv4 or IPv6 IP address range from an address pool.

   Example output

   BGP routing table entry for 203.0.113.200/30
   Paths: (1 available, best #1, table default)
     Advertised to non peer-group peers:
     10.0.2.3  1
     Local
       0.0.0.0 from 0.0.0.0 (10.0.1.2)
         Origin IGP, metric 0, weight 32768, valid, sourced, local, best (First path received)
         Last update: Mon Jan 10 19:49:07 2022

   1

   Confirm that the output includes an IP address for a BGP peer.

Procedure

1. Display the names of the speaker pods:

   $ oc get -n metallb-system pods -l component=speaker

   Example output

   NAME            READY   STATUS    RESTARTS   AGE
   speaker-66bth   4/4     Running   0          26m
   speaker-gvfnf   4/4     Running   0          26m
   ...

2. Display the BFD peers:

   $ oc exec -n metallb-system speaker-66bth -c frr -- vtysh -c "show bfd peers brief"

   Example output

   Session count: 2
   SessionId  LocalAddress              PeerAddress              Status
   =========  ============              ===========              ======
   3909139637 10.0.1.2                  10.0.2.3                 up  <.>

   <.> Confirm that the PeerAddress column includes each BFD peer. If the output does not list a BFD peer IP address that you expected the output to include, troubleshoot BGP connectivity with the peer. If the status field indicates down, check for connectivity on the links and equipment between the node and the peer. You can determine the node name for the speaker pod with a command like oc get pods -n metallb-system speaker-66bth -o jsonpath='{.spec.nodeName}'.