Networking

1. If unset, the default value is set to the next immediate subnet after the service network. For example, when the service network is 10.43.0.0/16, the advertiseAddress is set to 10.44.0.0/32.
2. You can use the multicast DNS protocol (mDNS) to allow name resolution and service discovery within a Local Area Network (LAN) using multicast exposed on the 5353/UDP port.
3. There is no built-in transparent proxying of egress traffic in MicroShift. Egress must be manually configured.
4. Setting up the firewalld service is supported by RHEL for Edge.
5. IPv6 is supported in both single-stack and dual-stack networks with the OVN-Kubernetes network plugin. IPv6 can also be used by connecting to other networks with the MicroShift Multus CNI plugin.
6. Configure by using the MicroShift config.yaml file.

Procedure

1. To create your ovn.yaml file, run the following command:

   $ sudo cp /etc/microshift/ovn.yaml.default /etc/microshift/ovn.yaml

2. To list the contents of the configuration file you created, run the following command:

   $ cat /etc/microshift/ovn.yaml

   Example YAML file with default maximum transmission unit (MTU) value

   mtu: 1400

3. To customize your configuration, you can change the MTU value. The table that follows provides details:

   Table 2.1. Supported optional OVN-Kubernetes configurations for MicroShift

   Field	Type	Default	Description	Example
   mtu	uint32	auto	MTU value used for the pods	1300

   Important

   If you change the mtu configuration value in the ovn.yaml file, you must restart the host that Red Hat build of MicroShift is running on to apply the updated setting.

   Example custom ovn.yaml configuration file

   mtu: 1300

1. Access the remote cluster by running the following command:

   $ export KUBECONFIG=$PWD/kubeconfig

2. Find the name of the ovnkube-master pod that you want to restart by running the following command:

   $ pod=$(oc get pods -n openshift-ovn-kubernetes | awk -F " " '/ovnkube-master/{print $1}')

3. Delete the ovnkube-master pod by running the following command:

   $ oc -n openshift-ovn-kubernetes delete pod $pod

4. Confirm that a new ovnkube-master pod is running by using the following command:

   $ oc get pods -n openshift-ovn-kubernetes

   The listing of the running pods shows a new ovnkube-master pod name and age.

Procedure

1. Add this setting to the /etc/systemd/system/rpm-ostreed.service.d/00-proxy.conf file:

   [Service]
   Environment="http_proxy=http://$PROXY_USER:$PROXY_PASSWORD@$PROXY_SERVER:$PROXY_PORT/"

2. Next, reload the configuration settings and restart the service to apply your changes.

   1. Reload the configuration settings by running the following command:

      $ sudo systemctl daemon-reload

   2. Restart the rpm-ostreed service by running the following command:

      $ sudo systemctl restart rpm-ostreed.service

Procedure

1. Create the directory for the configuration file if it does not exist:

   $ sudo mkdir /etc/systemd/system/crio.service.d/

2. Add the following settings to the /etc/systemd/system/crio.service.d/00-proxy.conf file:

   [Service]
   Environment=NO_PROXY="localhost,127.0.0.1"
   Environment=HTTP_PROXY="http://$PROXY_USER:$PROXY_PASSWORD@$PROXY_SERVER:$PROXY_PORT/"
   Environment=HTTPS_PROXY="http://$PROXY_USER:$PROXY_PASSWORD@$PROXY_SERVER:$PROXY_PORT/"

   Important

   You must define the Service section of the configuration file for the environment variables or the proxy settings fail to apply.

3. Reload the configuration settings:

   $ sudo systemctl daemon-reload

4. Restart the CRI-O service:

   $ sudo systemctl restart crio

5. Restart the MicroShift service to apply the settings:

   $ sudo systemctl restart microshift

Verification

1. Verify that pods are started by running the following command and examining the output:

   $ oc get all -A

2. Verify that MicroShift is able to pull container images by running the following command and examining the output:

   $ sudo crictl images

Procedure

1. Verify that your pods are running by entering the following command:

   $ oc get pods -A

   Example output

   NAMESPACE                            NAME                                                     READY   STATUS   RESTARTS  AGE
   default                              i-06166fbb376f14a8bus-west-2computeinternal-debug-qtwcr  1/1     Running	   0		   46m
   kube-system                          csi-snapshot-controller-5c6586d546-lprv4                 1/1     Running	   0		   51m
   kube-system                          csi-snapshot-webhook-6bf8ddc7f5-kz6k9                    1/1     Running	   0		   51m
   openshift-dns                        dns-default-45jl7                                        2/2     Running	   0		   50m
   openshift-dns                        node-resolver-7wmzf                                      1/1     Running	   0		   51m
   openshift-ingress                    router-default-78b86fbf9d-qvj9s                          1/1     Running 	 0		   51m
   openshift-multus                     dhcp-daemon-j7qnf                                        1/1     Running    0		   51m
   openshift-multus                     multus-r758z                                             1/1     Running    0		   51m
   openshift-operator-lifecycle-manager catalog-operator-85fb86fcb9-t6zm7                        1/1     Running    0		   51m
   openshift-operator-lifecycle-manager olm-operator-87656d995-fvz84                             1/1     Running    0		   51m
   openshift-ovn-kubernetes             ovnkube-master-5rfhh                                     4/4     Running    0		   51m
   openshift-ovn-kubernetes             ovnkube-node-gcnt6                                       1/1     Running    0		   51m
   openshift-service-ca                 service-ca-bf5b7c9f8-pn6rk                               1/1     Running    0		   51m
   openshift-storage                    topolvm-controller-549f7fbdd5-7vrmv                      5/5     Running    0		   51m
   openshift-storage                    topolvm-node-rht2m                                       3/3     Running    0		   50m

2. Create a namespace by running the following commands:

   $ NAMESPACE=<nginx-lb-test> 1

   1

   Replace _<nginx-lb-test> with the application namespace that you want to create.

   $ oc create ns $NAMESPACE

   Example namespace

   The following example deploys three replicas of the test nginx application in the created namespace:

   oc apply -n $NAMESPACE -f - <<EOF
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: nginx
   data:
     headers.conf: |
       add_header X-Server-IP  \$server_addr always;
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nginx
   spec:
     replicas: 3
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
         - image: quay.io/packit/nginx-unprivileged
           imagePullPolicy: Always
           name: nginx
           ports:
           - containerPort: 8080
           volumeMounts:
           - name: nginx-configs
             subPath: headers.conf
             mountPath: /etc/nginx/conf.d/headers.conf
           securityContext:
             allowPrivilegeEscalation: false
             seccompProfile:
               type: RuntimeDefault
             capabilities:
               drop: ["ALL"]
             runAsNonRoot: true
         volumes:
           - name: nginx-configs
             configMap:
               name: nginx
               items:
                 - key: headers.conf
                   path: headers.conf
   EOF

3. You can verify that the three sample replicas started successfully by running the following command:

   $ oc get pods -n $NAMESPACE

4. Create a LoadBalancer service for the nginx test application by running the following command:

   oc create -n $NAMESPACE -f - <<EOF
   apiVersion: v1
   kind: Service
   metadata:
     name: nginx
   spec:
     ports:
     - port: 81
       targetPort: 8080
     selector:
       app: nginx
     type: LoadBalancer
   EOF

   Note

   You must ensure that the port parameter is a host port that is not occupied by other LoadBalancer services or MicroShift components.

5. Verify that the service file exists, that the external IP address is properly assigned, and that the external IP is identical to the node IP by running the following command:

   $ oc get svc -n $NAMESPACE

   Example output

   NAME    TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
   nginx   LoadBalancer   10.43.183.104   192.168.1.241   81:32434/TCP   2m

Procedure

1. Change the NODEPORT variable to the host port number assigned to your Kubernetes NodePort service by running the following command:

   # export NODEPORT=30700

2. Change the INTERFACE_IP value to the IP address from the host interface that you want to block. For example:

   # export INTERFACE_IP=192.168.150.33

3. Insert a new rule in the nat table PREROUTING chain to drop all packets that match the destination port and IP address. For example:

   $ sudo nft -a insert rule ip nat PREROUTING tcp dport $NODEPORT ip daddr $INTERFACE_IP drop

4. List the new rule by running the following command:

   $ sudo nft -a list chain ip nat PREROUTING
   table ip nat {
   	chain PREROUTING { # handle 1
   		type nat hook prerouting priority dstnat; policy accept;
   		tcp dport 30700 ip daddr 192.168.150.33 drop # handle 134
   		counter packets 108 bytes 18074 jump OVN-KUBE-ETP # handle 116
   		counter packets 108 bytes 18074 jump OVN-KUBE-EXTERNALIP # handle 114
   		counter packets 108 bytes 18074 jump OVN-KUBE-NODEPORT # handle 112
   	}
   }

   Note

   Note the handle number of the newly added rule. You need to remove the handle number in the following step.

5. Remove the custom rule with the following sample command:

   $ sudo nft -a delete rule ip nat PREROUTING handle 134

Procedure

1. Update the value of ingress.status field to Removed in the MicroShift config.yaml file as shown in the following example:

   Example config.yaml ingress stanza

   # ...
   ingress:
     ports:
       http: 80
       https: 443
     routeAdmissionPolicy:
       namespaceOwnership: InterNamespaceAllowed
     status: Removed 1
   # ...

   1

   When the value is set to Removed, the ports listed in ingress.ports are automatically closed. Any other settings in the ingress stanza are ignored, for example, any values in the routeAdmissionPolicy.namespaceOwnership field.

2. Restart the MicroShift service by running the following command:

   $ sudo systemctl restart microshift

   Note

   The MicroShift service outputs current configurations during restarts.

Procedure

1. To prevent routes in different namespaces from claiming the same hostname, update the namespaceOwnership field value to Strict in the MicroShift config.yaml file. See the following example:

   Example config.yaml route admission policy

   # ...
   ingress:
     routeAdmissionPolicy:
       namespaceOwnership: Strict 1
   # ...

   1

   Prevents routes in different namespaces from claiming the same host. Valid values are Strict and InterNamespaceAllowed. If you delete the value in a customized config.yaml, the InterNamespaceAllowed value is set automatically.

2. To apply the configuration, restart the MicroShift service by running the following command:

   $ sudo systemctl restart microshift

Procedure

1. Create a service called hello-microshift by running the following command:

   $ oc expose pod hello-microshift -n $namespace

2. Create an unsecured route to the hello-microshift application by running the following command:

   $ oc expose svc/hello-microshift --hostname=microshift.com $namespace

Procedure

1. Create a route definition and save it in a file called app-example-route.yaml:

   YAML definition of the created route with HTTP header directives

   apiVersion: route.openshift.io/v1
   kind: Route
   # ...
   spec:
     host: app.example.com
     tls:
       termination: edge
     to:
       kind: Service
       name: app-example
     httpHeaders:
       actions: 1
         response: 2
         - name: Content-Location 3
           action:
             type: Set 4
             set:
               value: /lang/en-us 5

   1

   The list of actions you want to perform on the HTTP headers.

   2

   The type of header you want to change. In this case, a response header.

   3

   The name of the header you want to change. For a list of available headers you can set or delete, see HTTP header configuration.

   4

   The type of action being taken on the header. This field can have the value Set or Delete.

   5

   When setting HTTP headers, you must provide a value. The value can be a string from a list of available directives for that header, for example DENY, or it can be a dynamic value that will be interpreted using HAProxy’s dynamic value syntax. In this case, the value is set to the relative location of the content.

2. Create a route to your existing web application using the newly created route definition:

   $ oc -n app-example create -f app-example-route.yaml

Procedure

1. Define an Ingress object in the Red Hat build of MicroShift console or by entering the oc create command:

   YAML Definition of an Ingress

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: frontend
     annotations:
       route.openshift.io/termination: "reencrypt" 1
       route.openshift.io/destination-ca-certificate-secret: secret-ca-cert 2
   spec:
     rules:
     - host: www.example.com 3
       http:
         paths:
         - backend:
             service:
               name: frontend
               port:
                 number: 443
           path: /
           pathType: Prefix
     tls:
     - hosts:
       - www.example.com
       secretName: example-com-tls-certificate

   1

   The route.openshift.io/termination annotation can be used to configure the spec.tls.termination field of the Route as Ingress has no field for this. The accepted values are edge, passthrough and reencrypt. All other values are silently ignored. When the annotation value is unset, edge is the default route. The TLS certificate details must be defined in the template file to implement the default edge route.

   3

   When working with an Ingress object, you must specify an explicit hostname, unlike when working with routes. You can use the <host_name>.<cluster_ingress_domain> syntax, for example apps.openshiftdemos.com, to take advantage of the *.<cluster_ingress_domain> wildcard DNS record and serving certificate for the cluster. Otherwise, you must ensure that there is a DNS record for the chosen hostname.

   1. If you specify the passthrough value in the route.openshift.io/termination annotation, set path to '' and pathType to ImplementationSpecific in the spec:

        spec:
          rules:
          - host: www.example.com
            http:
              paths:
              - path: ''
                pathType: ImplementationSpecific
                backend:
                  service:
                    name: frontend
                    port:
                      number: 443

      $ oc apply -f ingress.yaml

   2

   The route.openshift.io/destination-ca-certificate-secret can be used on an Ingress object to define a route with a custom destination certificate (CA). The annotation references a kubernetes secret, secret-ca-cert that will be inserted into the generated route.

   1. To specify a route object with a destination CA from an ingress object, you must create a kubernetes.io/tls or Opaque type secret with a certificate in PEM-encoded format in the data.tls.crt specifier of the secret.

2. List your routes:

   $ oc get routes

   The result includes an autogenerated route whose name starts with frontend-:

   NAME             HOST/PORT         PATH    SERVICES    PORT    TERMINATION          WILDCARD
   frontend-gnztq   www.example.com           frontend    443     reencrypt/Redirect   None

   If you inspect this route, it looks this:

   YAML Definition of an autogenerated route

   apiVersion: route.openshift.io/v1
   kind: Route
   metadata:
     name: frontend-gnztq
     ownerReferences:
     - apiVersion: networking.k8s.io/v1
       controller: true
       kind: Ingress
       name: frontend
       uid: 4e6c59cc-704d-4f44-b390-617d879033b6
   spec:
     host: www.example.com
     path: /
     port:
       targetPort: https
     tls:
       certificate: |
         -----BEGIN CERTIFICATE-----
         [...]
         -----END CERTIFICATE-----
       insecureEdgeTerminationPolicy: Redirect
       key: |
         -----BEGIN RSA PRIVATE KEY-----
         [...]
         -----END RSA PRIVATE KEY-----
       termination: reencrypt
       destinationCACertificate: |
         -----BEGIN CERTIFICATE-----
         [...]
         -----END CERTIFICATE-----
     to:
       kind: Service
       name: frontend

Procedure

1. Create a YAML file for the Ingress object. In this example, the file is called example-ingress.yaml:

   YAML definition of an Ingress object

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: frontend
     ...
   spec:
     rules:
       ...
     tls:
     - {} 1

   1

   Use this exact syntax to specify TLS without specifying a custom certificate.

2. Create the Ingress object by running the following command:

   $ oc create -f example-ingress.yaml

Procedure

1. Add the route.openshift.io/destination-ca-certificate-secret to the Ingress annotations:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: frontend
     annotations:
       route.openshift.io/termination: "reencrypt"
       route.openshift.io/destination-ca-certificate-secret: secret-ca-cert 1
   ...

   1

   The annotation references a kubernetes secret.

2. The secret referenced in this annotation will be inserted into the generated route.

   Example output

   apiVersion: route.openshift.io/v1
   kind: Route
   metadata:
     name: frontend
     annotations:
       route.openshift.io/termination: reencrypt
       route.openshift.io/destination-ca-certificate-secret: secret-ca-cert
   spec:
   ...
     tls:
       insecureEdgeTerminationPolicy: Redirect
       termination: reencrypt
       destinationCACertificate: |
         -----BEGIN CERTIFICATE-----
         [...]
         -----END CERTIFICATE-----
   ...

Procedure

1. Optional: Check for firewalld on your system by running the following command:

   $ rpm -q firewalld

2. If the firewalld service is not installed, run the following command:

   $ sudo dnf install -y firewalld

3. To start the firewall, run the following command:

   $ sudo systemctl enable firewalld --now

Procedure

1. Optional: You can view all predefined services in firewalld by running the following command

   $ sudo firewall-cmd --get-services

2. To open a service that you want on a default port, run the following example command:

   $ sudo firewall-cmd --add-service=mdns

Procedure

1. Use one of the following commands to set the IP address range:

   1. Configure the IP address range with default values by running the following command:

      $ sudo firewall-offline-cmd --permanent --zone=trusted --add-source=10.42.0.0/16

   2. Configure the IP address range with custom values by running the following command:

      $ sudo firewall-offline-cmd --permanent --zone=trusted --add-source=<custom IP range>

2. To allow internal traffic from pods through the network gateway, run the following command:

   $ sudo firewall-offline-cmd --permanent --zone=trusted --add-source=169.254.169.1

Procedure

1. Stop the MicroShift service by running the following command:

   $ sudo systemctl stop microshift

2. Stop the kubepods.slice systemd unit by running the following command:

   $ sudo systemctl stop kubepods.slice

3. MicroShift installs a helper script to undo network changes made by OVN-K. Run the cleanup script by entering the following command:

   $ sudo /usr/bin/microshift-cleanup-data --ovn

Procedure

1. Add a fake IP address to the loopback interface by running the following command:

   $ IP="10.44.0.1" 1
   $ sudo nmcli con add type loopback con-name stable-microshift ifname lo ip4 ${IP}/32

   1

   The fake IP address used in this example is “10.44.0.1”.

   Note

   Any valid IP works if it avoids both internal MicroShift and potential future external IP conflicts. This can be any subnet that does not collide with the MicroShift node subnet or is be accessed by other services on the device.

2. Configure the DNS interface to use the local name server by setting modifying the settings to ignore automatic DNS and reset it to the local name server:

   1. Bypass the automatic DNS by running the following command:

      $ sudo nmcli conn modify stable-microshift ipv4.ignore-auto-dns yes

   2. Point the DNS interface to use the local name server:

      $ sudo nmcli conn modify stable-microshift ipv4.dns "10.44.1.1"

3. Get the hostname of the device by running the following command:

   $ NAME="$(hostnamectl hostname)"

4. Add an entry for the hostname of the node in the /etc/hosts file by running the following command:

   $ echo "$IP $NAME" | sudo tee -a /etc/hosts >/dev/null

5. Update the MicroShift configuration file by adding the following YAML snippet to /etc/microshift/config.yaml:

   sudo tee /etc/microshift/config.yaml > /dev/null <<EOF
   node:
     hostnameOverride: $(echo $NAME)
     nodeIP: $(echo $IP)
   EOF

6. MicroShift is now ready to use the loopback device for cluster communications. Finish preparing the device for offline use.

   1. If the device currently has a NIC attached, disconnect the device from the network.
   2. Shut down the device and disconnect the NIC.
   3. Restart the device for the offline configuration to take effect.

7. Restart the MicroShift host to apply the configuration changes by running the following command:

   $ sudo systemctl reboot 1

   1

   This step restarts the cluster. Wait for the greenboot health check to report the system healthy before implementing verification.

1. Verify that the MicroShift cluster is running by entering the following command:

   $ export KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig
   $ sudo -E oc get pods -A

   Example output

   NAMESPACE                  NAME                                       READY   STATUS    RESTARTS      AGE
   kube-system                csi-snapshot-controller-74d566564f-66n2f   1/1     Running   0             1m
   kube-system                csi-snapshot-webhook-69bdff8879-xs6mb      1/1     Running   0             1m
   openshift-dns              dns-default-dxglm                          2/2     Running   0             1m
   openshift-dns              node-resolver-dbf5v                        1/1     Running   0             1m
   openshift-ingress          router-default-8575d888d8-xmq9p            1/1     Running   0             1m
   openshift-ovn-kubernetes   ovnkube-master-gcsx8                       4/4     Running   1             1m
   openshift-ovn-kubernetes   ovnkube-node-757mf                         1/1     Running   1             1m
   openshift-service-ca       service-ca-7d7c579f54-68jt4                1/1     Running   0             1m
   openshift-storage          topolvm-controller-6d777f795b-bx22r        5/5     Running   0             1m
   openshift-storage          topolvm-node-fcf8l                         4/4     Running   0             1m