Configuring

Procedure

1. If you have not done so, make a copy of the provided config.yaml.default file in the /etc/microshift/ directory, renaming it config.yaml.

2. Keep the new MicroShift config.yaml in the /etc/microshift/ directory. Your config.yaml file is read every time the MicroShift service starts.

   Note

   After you create it, the config.yaml file takes precedence over built-in settings.

3. Replace the default values in the network section of the MicroShift YAML with your valid values.

   Example single-stack IPv6 networking configuration

   apiServer:
   # ...
   network:
     clusterNetwork:
     - fd01::/48 

   1

   
     serviceNetwork:
     - fd02::/112 

   2

   
   node:
     nodeIP: 2600:1f14:1c48:ee00:2d76:3190:5bc2:5aef 

   3

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Specify a clusterNetwork with a CIDR value that is less than 64.

   2

   Specify an IPv6 CIDR with a prefix of 112. Kubernetes uses only the lowest 16 bits. For a prefix of 112, IP addresses are assigned from 112 to 128 bits.

   3

   Example node IP address. Valid values are IP addresses in the IPv6 address family. You must only specify an IPv6 address when an IPv4 network is also present. If an IPv4 network is not present, the MicroShift service automatically fills in this value upon restart.

4. Complete any other configurations you require, then start MicroShift by running the following command:

   $ sudo systemctl start microshift

   Copy to Clipboard Toggle word wrap

Verification

1. Retrieve the networks defined in the node resource by running the following command:

   $ oc get node -o jsonpath='{.items[].spec.podCIDRs[]}'

   Copy to Clipboard Toggle word wrap

   Example output

   fd01::/48

   Copy to Clipboard Toggle word wrap

2. Retrieve the status of the pods by running the following command:

   $ oc get pod -A -o wide

   Copy to Clipboard Toggle word wrap

   Example output

   NAMESPACE                  NAME                                      READY   STATUS    RESTARTS   AGE   IP                      NODE           NOMINATED NODE   READINESS GATES
   kube-system                csi-snapshot-controller-bb7cb654b-rqrt6   1/1     Running   0          65s   fd01:0:0:1::5           microshift-9   <none>           <none>
   openshift-dns              dns-default-cjn66                         2/2     Running   0          62s   fd01:0:0:1::9           microshift-9   <none>           <none>
   openshift-dns              node-resolver-ppnjb                       1/1     Running   0          63s   2001:db9:ca7:ff::1db8   microshift-9   <none>           <none>
   openshift-ingress          router-default-6d97d7b8b6-wdtmg           1/1     Running   0          61s   fd01:0:0:1::8           microshift-9   <none>           <none>
   openshift-ovn-kubernetes   ovnkube-master-gfvp5                      4/4     Running   0          63s   2001:db9:ca7:ff::1db8   microshift-9   <none>           <none>
   openshift-ovn-kubernetes   ovnkube-node-bnpjh                        1/1     Running   0          63s   2001:db9:ca7:ff::1db8   microshift-9   <none>           <none>
   openshift-service-ca       service-ca-5d7bd9db6-j25bd                1/1     Running   0          60s   fd01:0:0:1::4           microshift-9   <none>           <none>
   openshift-storage          lvms-operator-656cd9b59b-bwr47            1/1     Running   0          63s   fd01:0:0:1::7           microshift-9   <none>           <none>
   openshift-storage          vg-manager-f7dmk                          1/1     Running   0          27s   fd01:0:0:1::a           microshift-9   <none>           <none>

   Copy to Clipboard Toggle word wrap

3. Retrieve the status of services by running the following command:

   $ oc get svc -A

   Copy to Clipboard Toggle word wrap

   Example output

   NAMESPACE           NAME                            TYPE           CLUSTER-IP   EXTERNAL-IP                                             PORT(S)                      AGE
   default             kubernetes                      ClusterIP      fd02::1      <none>                                                  443/TCP                      3m42s
   openshift-dns       dns-default                     ClusterIP      fd02::a      <none>                                                  53/UDP,53/TCP,9154/TCP       2m58s
   openshift-ingress   router-default                  LoadBalancer   fd02::f2e6   2001:db9:ca7:ff::1db8,fd01:0:0:1::2,fd02::1:0,fd69::2   80:31133/TCP,443:31996/TCP   2m58s
   openshift-ingress   router-internal-default         ClusterIP      fd02::c55e   <none>                                                  80/TCP,443/TCP,1936/TCP      2m58s
   openshift-storage   lvms-operator-metrics-service   ClusterIP      fd02::7afb   <none>                                                  443/TCP                      2m58s
   openshift-storage   lvms-webhook-service            ClusterIP      fd02::d8dd   <none>                                                  443/TCP                      2m58s
   openshift-storage   vg-manager-metrics-service      ClusterIP      fd02::fc1    <none>                                                  443/TCP                      2m58s

   Copy to Clipboard Toggle word wrap

Procedure

1. If you have not done so, make a copy of the provided config.yaml.default file in the /etc/microshift/ directory, renaming it config.yaml.

2. Keep the new MicroShift config.yaml in the /etc/microshift/ directory. Your config.yaml file is read every time the MicroShift service starts.

   Note

   After you create it, the config.yaml file takes precedence over built-in settings.

3. If you have not started MicroShift, replace the default values in the network section of the MicroShift YAML with your valid values.

   Example dual-stack IPv6 networking configuration with network assignments

   apiServer:
   # ...
   apiServer:
     subjectAltNames:
     - 192.168.113.117
     - 2001:db9:ca7:ff::1db8
   network:
     clusterNetwork:
     - 10.42.0.0/16
     - fd01::/48 

   1

   
     serviceNetwork:
     - 10.43.0.0/16
     - fd02::/112 

   2

   
   node:
     nodeIP: 192.168.113.117 

   3

   
     nodeIPv6: 2001:db9:ca7:ff::1db8 

   4

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Specify an IPv6 clusterNetwork with a CIDR value that is less than 64.

   2

   Specify an IPv6 CIDR with a prefix of 112. Kubernetes uses only the lowest 16 bits. For a prefix of 112, IP addresses are assigned from 112 to 128 bits.

   3

   Example node IP address. Must be an IPv4 address family.

   4

   Example node IP address for dual-stack configuration. Must be an IPv6 address family. Configurable only with dual-stack networking.

4. Complete any other MicroShift configurations you require, then start MicroShift by running the following command:

   $ sudo systemctl start microshift

   Copy to Clipboard Toggle word wrap
5. Reset the IP family policy for application pods and services as needed, then restart those application pods and services to enable dual-stack networking. See "Resetting the IP family policy for application pods and services" for a simple example.

Verification

1. You can verify that all of the system services and pods to have two IP addresses, one for each family, by using the following steps:

   1. Retrieve the networks defined in the node resource by running the following command:

      $ oc get pod -n openshift-ingress router-default-5b75594b4-w7w6s -o jsonpath='{.status.podIPs}'

      Copy to Clipboard Toggle word wrap

      Example output

      [{"ip":"10.42.0.4"},{"ip":"fd01:0:0:1::4"}]

      Copy to Clipboard Toggle word wrap

   2. Retrieve the networks defined by the host network pods by running the following command:

      $ oc get pod -n openshift-ovn-kubernetes ovnkube-master-2fm2k -o jsonpath='{.status.podIPs}'

      Copy to Clipboard Toggle word wrap

      Example output

      [{"ip":"192.168.113.117"},{"ip":"2001:db9:ca7:ff::1db8"}]

      Copy to Clipboard Toggle word wrap

Procedure

1. If you have not done so, make a copy of the provided config.yaml.default file in the /etc/microshift/ directory, renaming it config.yaml.

2. Keep the new MicroShift config.yaml in the /etc/microshift/ directory. Your config.yaml file is read every time the MicroShift service starts.

   Note

   After you create it, the config.yaml file takes precedence over built-in settings.

3. Add IPv6 configurations to the network section of the MicroShift YAML with your valid values:

   Warning

   You must keep the same first entry across restarts and migrations. This is true for any migration: single-to-dual stack, or dual-to-single stack. A complete wipe of the etcd database is required if a change to the first entry is needed. This might result in application data loss and is not supported.

   1. Add an IPv6 configuration for a second network in the network section of the MicroShift YAML with your valid values.

   2. Add network assignments to the network section of the MicroShift config.yaml to enable dual stack with IPv6 as secondary network.

      Example dual-stack IPv6 configuration with network assignments

      # ...
      apiServer:
        subjectAltNames:
        - 192.168.113.117
        - 2001:db9:ca7:ff::1db8 

      1

      
      network:
        clusterNetwork:
        - 10.42.0.0/16 

      2

      
        - fd01::/48 

      3

      
        serviceNetwork:
        - 10.43.0.0/16
        - fd02::/112 

      4

      
      node:
        nodeIP: 192.168.113.117 

      5

      
        nodeIPv6: 2001:db9:ca7:ff::1db8 

      6

      
      # ...

      Copy to Clipboard Toggle word wrap

      1

      The IPv6 node address.

      2

      IPv4 network. Specify a clusterNetwork with a CIDR value that is less than 24.

      3

      IPv6 network. Specify a clusterNetwork with a CIDR value that is less than 64.

      4

      Specify an IPv6 CIDR with a prefix of 112. Kubernetes uses only the lowest 16 bits. For a prefix of 112, IP addresses are assigned from 112 to 128 bits.

      5

      Example node IP address. Maintain the previous IPv4 IP address.

      6

      Example node IP address. Must be an IPv6 address family.

4. Complete any other configurations you require, then restart MicroShift by running the following command:

   $ sudo systemctl restart microshift

   Copy to Clipboard Toggle word wrap
5. Reset the IP family policy for application pods and services as needed, then restart those application pods and services to enable dual-stack networking. See "Resetting the IP family policy for application pods and services" for a simple example.

1. Retrieve the status of the pods by running the following command:

   $ oc get pod -A -o wide

   Copy to Clipboard Toggle word wrap

   Example output

   NAMESPACE                  NAME                                      READY   STATUS    RESTARTS        AGE     IP                NODE           NOMINATED NODE   READINESS GATES
   kube-system                csi-snapshot-controller-bb7cb654b-7s5ql   1/1     Running   0               46m     10.42.0.6         microshift-9   <none>           <none>
   openshift-dns              dns-default-zxkqn                         2/2     Running   0               46m     10.42.0.5         microshift-9   <none>           <none>
   openshift-dns              node-resolver-r2h5z                       1/1     Running   0               46m     192.168.113.117   microshift-9   <none>           <none>
   openshift-ingress          router-default-5b75594b4-228z7            1/1     Running   0               2m5s    10.42.0.3         microshift-9   <none>           <none>
   openshift-ovn-kubernetes   ovnkube-master-bltk7                      4/4     Running   2 (2m32s ago)   2m36s   192.168.113.117   microshift-9   <none>           <none>
   openshift-ovn-kubernetes   ovnkube-node-9ghgs                        1/1     Running   2 (2m32s ago)   46m     192.168.113.117   microshift-9   <none>           <none>
   openshift-service-ca       service-ca-5d7bd9db6-qgwgw                1/1     Running   0               46m     10.42.0.7         microshift-9   <none>           <none>
   openshift-storage          lvms-operator-656cd9b59b-8rpf4            1/1     Running   0               46m     10.42.0.8         microshift-9   <none>           <none>
   openshift-storage          vg-manager-wqmh4                          1/1     Running   2 (2m39s ago)   46m     10.42.0.10        microshift-9   <none>           <none>

   Copy to Clipboard Toggle word wrap

2. Retrieve the networks defined by the OVN-K network plugin by running the following command:

   $ oc get pod -n openshift-ovn-kubernetes ovnkube-master-bltk7 -o jsonpath='{.status.podIPs}'

   Copy to Clipboard Toggle word wrap

   Example output

   [{"ip":"192.168.113.117"},{"ip":"2001:db9:ca7:ff::1db8"}]

   Copy to Clipboard Toggle word wrap

3. Retrieve the networks defined in the node resource by running the following command:

   $ oc get pod -n openshift-ingress router-default-5b75594b4-228z7 -o jsonpath='{.status.podIPs}'

   Copy to Clipboard Toggle word wrap

   Example output

   [{"ip":"10.42.0.3"},{"ip":"fd01:0:0:1::3"}]

   Copy to Clipboard Toggle word wrap

Procedure

1. Set the spec.ipFamilyPolicy field to a valid value for dual-stack networking in your service or pod by using the following example:

   Example dual-stack network configuration for a service

   kind: Service
   apiVersion: v1
   metadata:
     name: microshift-new-service
     labels: app: microshift-application
   spec:
     type: NodePort
     ipFamilyPolicy: `PreferDualStack` 

   1

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Required. Valid values for dual-stack networking are PreferDualStack and RequireDualStack. The value you set depends on the requirements of your application. PreferSingleStack is the default value for the ipFamilyPolicy field.

2. Restart any application pods that do not have a hostNetwork defined. Pods that do have a hostNetwork defined do not need to be restarted to update the ipFamilyPolicy value.

Procedure

1. Apply ingress control settings in one of the two following ways:

   1. Update the MicroShift config.yaml configuration file by making a copy of the provided config.yaml.default file in the /etc/microshift/ directory, naming it config.yaml and keeping it in the source directory.

      • After you create it, the config.yaml file takes precedence over built-in settings. The configuration file is read every time the MicroShift service starts.

   2. Use a configuration snippet to apply the ingress control settings you want. To do this, create a configuration snippet YAML file and put it in the /etc/microshift/config.d/ configuration directory.

      • Configuration snippet YAMLs take precedence over both built-in settings and a config.yaml configuration file. See the Additional resources links for more information.

2. Replace the default values in the network section of the MicroShift YAML with your valid values, or create a configuration snippet file with the sections you need.

   Ingress controller configuration fields with default values

   apiServer:
   # ...
   ingress:
       defaultHTTPVersion: 1
       forwardedHeaderPolicy: Append
       httpCompression:
           mimeTypes:
               - ""
       httpEmptyRequestsPolicy: Respond
   # ...
       logEmptyRequests: Log
   # ...
       tuningOptions:
           clientFinTimeout: 1s
           clientTimeout: 30s
           headerBufferBytes: 0
           headerBufferMaxRewriteBytes: 0
           healthCheckInterval: 5s
           maxConnections: 0
           serverFinTimeout: 1s
           serverTimeout: 30s
           threadCount: 4
           tlsInspectDelay: 5s
           tunnelTimeout: 1h
   # ...

   Copy to Clipboard Toggle word wrap

   Expand

   Table 3.1. Ingress controller configuration fields definitions table

   Parameter	Description
   defaultHTTPVersion	Sets the HTTP version for the ingress controller. Default value is 1 for HTTP 1.1.
   forwardedHeaderPolicy	Specifies when and how the ingress controller sets the Forwarded, X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Port, X-Forwarded-Proto, and X-Forwarded-Proto-Version HTTP headers. The following values are valid: Append, preserves any existing headers by specifying that the ingress controller appends them. Replace, removes any existing headers by specifying that the ingress controller sets the headers. IfNone sets the headers set by specifying that the ingress controller sets the headers if they are not already set. Never, preserves any existing headers by specifying that the ingress controller never sets the headers.
   httpCompression	Defines the policy for HTTP traffic compression. httpCompression.mimeTypes defines a list of or a single MIME type to which compression is applied. For example, text/css; charset=utf-8, text/html, text/*, image/svg+xml, application/octet-stream, X-custom/customsub, using the format pattern, type/subtype; [;attribute=value]. The types are: application, image, message, multipart, text, video, or a custom type prefaced by X-. To see the full notation for MIME types and subtypes, see RFC1341 (IETF Datatracker documentation).
   httpEmptyRequestsPolicy	Describes how HTTP connections are handled if the connection times out before a request is received. Allowed values for this field are Respond and Ignore. The default value is Respond. The following are valid values: Respond, causes the ingress controller to send an HTTP 400 or 408 response, logs the connection if access logging is enabled, and counts the connection in the appropriate metrics. Ignore, adds the http-ignore-probes parameter in the HAproxy configuration. If the field is set to Ignore, the ingress controller closes the connection without sending a response, then logs the connection or incrementing metrics. Usually, empty request connections come from load balancer health probes or web browser preconnects and can be safely ignored. However, network errors and port scans can also create these empty requests, so setting this field to Ignore can impede detecting or diagnosing problems and also impede the detection of intrusion attempts.
   logEmptyRequests	Specifies connections for which no request is received and logged. Usually, these empty requests come from load balancer health probes or web browser speculative connections such as preconnects. Logging these types of empty requests can be undesirable. However, network errors and port scans can also create empty requests, so setting this field to Ignore can impede detecting or diagnosing problems and also impede the detection of intrusion attempts. The following are valid values: Log, which indicates that an event should be logged. Ignore, which sets the dontlognull option in the HAproxy configuration.
   tuningOptions	Specifies options for tuning the performance of ingress controller pods. The tuningOptions.clientFinTimeout parameter specifies how long a connection is held open while waiting for the client response to the server closing the connection. The default timeout is 1s. The tuningOptions.clientTimeout parameter specifies how long a connection is held open while waiting for a client response. The default timeout is 30s. The tuningOptions.headerBufferBytes parameter specifies how much memory is reserved, in bytes, for Ingress Controller connection sessions. This value must be at least 16384 if HTTP/2 is enabled for the Ingress Controller. If not set, the default value is 32768 bytes. Important Setting this field not recommended because headerBufferMaxRewriteBytes parameter values that are too small can break the ingress controller. Conversely, values for headerBufferMaxRewriteBytes that are too large could cause the ingress controller to use significantly more memory than necessary. The tuningOptions.headerBufferMaxRewriteBytes parameter specifies how much memory should be reserved, in bytes, from headerBufferBytes for HTTP header rewriting and appending for Ingress Controller connection sessions. The minimum value for headerBufferMaxRewriteBytes is 4096. The headerBufferBytes value must be greater than the headerBufferMaxRewriteBytes value for incoming HTTP requests. If not set, the default value is 8192 bytes. Important Setting this field is not recommended because headerBufferMaxRewriteBytes parameter values that are too small can break the ingress controller. Conversely, values for headerBufferMaxRewriteBytes that are too large could cause the ingress controller to use significantly more memory than necessary. The tuningOptions.healthCheckInterval parameter specifies how long the router waits between health checks. The default is 5s. The tuningOptions.serverFinTimeout parameter specifies how long a connection is held open while waiting for the server response to the client that is closing the connection. The default timeout is 1s. The tuningOptions.serverTimeout parameter specifies how long a connection is held open while waiting for a server response. The default timeout is 30s. The tuningOptions.threadCount parameter specifies the number of threads to create per HAProxy process. Creating more threads allows each ingress controller pod to handle more connections, at the cost of more system resources being used. HAProxy supports up to 64 threads. If this field is empty, default value is 4 threads. Important Setting this field is not recommended because increasing the number of HAProxy threads allows ingress controller pods to use more CPU time under load, and prevent other pods from receiving the CPU resources they need to perform. The tuningOptions.tlsInspectDelay parameter specifies how long the router can hold data to find a matching route. Setting this value too short can cause the router to fall back to the default certificate for edge-terminated, re-encrypted, or passthrough routes, even when using a better-matched certificate. The default inspect delay is 5s. The tuningOptions.tunnelTimeout parameter specifies how long a tunnel connection, including websockets, remains open while the tunnel is idle. The default timeout is 1h.

   Show more

3. Complete any other configurations you require, then start or restart MicroShift by running one the following commands:

   $ sudo systemctl start microshift

   Copy to Clipboard Toggle word wrap

   $ sudo systemctl restart microshift

   Copy to Clipboard Toggle word wrap

Procedure

1. Optional: You can show the contents of the config.yaml. Run the following command:

   $ cat /etc/microshift/config.yaml

   Copy to Clipboard Toggle word wrap

2. Optional: You can show the contents of the remote-access kubeconfig file. Run the following command:

   $ cat /var/lib/microshift/resources/kubeadmin/<hostname>/kubeconfig

   Copy to Clipboard Toggle word wrap
   Important

   Additional remote access kubeconfig files must include one of the server names listed in the Red Hat build of MicroShift config.yaml file. Additional kubeconfig files must also use the same CA for validation.

3. To generate additional kubeconfig files for additional DNS names SANs or external IP addresses, add the entries you need to the apiServer.subjectAltNames field. In the following example, the DNS name used is alt-name-1 and the IP address is 1.2.3.4.

   Example config.yaml with additional authentication values

   dns:
     baseDomain: example.com
   node:
     hostnameOverride: "microshift-rhel9" 

   1

   
     nodeIP: 10.0.0.1
   apiServer:
     subjectAltNames:
     - alt-name-1 

   2

   
     - 1.2.3.4 

   3

   Copy to Clipboard Toggle word wrap

   1

   Hostname

   2

   DNS name

   3

   IP address or range

4. Restart MicroShift to apply configuration changes and auto-generate the kubeconfig files you need by running the following command:

   $ sudo systemctl restart microshift

   Copy to Clipboard Toggle word wrap

5. To check the contents of additional remote-access kubeconfig files, insert the name or IP address as listed in the config.yaml into the cat command. For example, alt-name-1 is used in the following example command:

   $ cat /var/lib/microshift/resources/kubeadmin/alt-name-1/kubeconfig

   Copy to Clipboard Toggle word wrap

6. Choose the kubeconfig file to use that contains the SAN or IP address you want to use to connect your cluster. In this example, the kubeconfig containing`alt-name-1` in the cluster.server field is the correct file.

   Example contents of an additional kubeconfig file

   clusters:
   - cluster:
       certificate-authority-data: <base64 CA>
       server: https://alt-name-1:6443 

   1

   Copy to Clipboard Toggle word wrap

   1

   The /var/lib/microshift/resources/kubeadmin/alt-name-1/kubeconfig file values are from the apiServer.subjectAltNames configuration values.

1. Copy the certificates and keys to the preferred directory in the host operating system. Ensure that the files are accessible by root only.

2. Update the MicroShift configuration for each custom CA by specifying the certificate names and new fully qualified domain name (FQDN) in the MicroShift /etc/microshift/config.yaml configuration file.

   Each certificate configuration can contain the following values:

   • The certificate file location is a required value.

   • A single common name containing the API server DNS and IP address or IP address range.

     Tip

     In most cases, MicroShift generates a new kubeconfig for your custom CA that includes the IP address or range that you specify. The exception is when wildcards are specified for the IP address. In this case, MicroShift generates a kubeconfig with the public IP address of the server. To use wildcards, you must update the kubeconfig file with your specific details.

   • Multiple Subject Alternative Names (SANs) containing the API server DNS and IP addresses or a wildcard certificate.
   • You can provide additional DNS names for each certificate.
3. After the MicroShift service restarts, you must copy the generated kubeconfig files to the client.
4. Configure additional CAs on the client system. For example, you can update CA bundles in the Red Hat Enterprise Linux (RHEL) truststore.
5. The certificates and keys are read from the specified file location on the host. Testing and validation of configuration is done from the client.
6. External server certificates are not automatically renewed. You must manually rotate your external certificates.

Procedure

1. Copy the custom certificates you want to add to the trust root of the MicroShift host. Ensure that the certificate and private keys are only accessible to MicroShift.

2. For each custom CA that you need, add an apiServer section called namedCertificates to the /etc/microshift/config.yaml MicroShift configuration file by using the following example:

   apiServer:
     namedCertificates:
      - certPath: ~/certs/api_fqdn_1.crt 

   1

   
        keyPath:  ~/certs/api_fqdn_1.key 

   2

   
      - certPath: ~/certs/api_fqdn_2.crt
        keyPath:  ~/certs/api_fqdn_2.key
        names: 

   3

   
        - api_fqdn_1
        - *.apps.external.com

   Copy to Clipboard Toggle word wrap

   1

   Add the full path to the certificate.

   2

   Add the full path to the certificate key.

   3

   Optional. Add a list of explicit DNS names. Leading wildcards are allowed. If no names are provided, the implicit names are extracted from the certificates.

3. Restart the MicroShift to apply the certificates by running the following command:

   $ systemctl microshift restart

   Copy to Clipboard Toggle word wrap
4. Wait a few minutes for the system to restart and apply the custom server. New kubeconfig files are generated in the /var/lib/microshift/resources/kubeadmin/ directory.
5. Copy the kubeconfig files to the client. If you specified wildcards for the IP address, update the kubeconfig to remove the public IP address of the server and replace that IP address with the specific wildcard range you want to use.

6. From the client, use the following steps:

   1. Specify the kubeconfig to use by running the following command:

      $ export KUBECONFIG=~/custom-kubeconfigs/kubeconfig 

      1

      Copy to Clipboard Toggle word wrap

      1

      Use the location of the copied kubeconfig file as the path.

   2. Check that the certificates are applied by using the following command:

      $ oc --certificate-authority ~/certs/ca.ca get node

      Copy to Clipboard Toggle word wrap

      Example output

      oc get node
      NAME                             STATUS   ROLES                         AGE   VERSION
      dhcp-1-235-195.arm.example.com   Ready    control-plane,master,worker   76m   v1.31.3

      Copy to Clipboard Toggle word wrap

   3. Add the new CA file to the $KUBECONFIG environment variable by running the following command:

      $ oc config set clusters.microshift.certificate-authority /tmp/certificate-authority-data-new.crt

      Copy to Clipboard Toggle word wrap

   4. Verify that the new kubeconfig file contains the new CA by running the following command:

      $ oc config view --flatten

      Copy to Clipboard Toggle word wrap

      Example externally generated kubeconfig file

      apiVersion: v1
      clusters:
      - cluster:
          certificate-authority: /tmp/certificate-authority-data-new.crt 

      1

      
          server: https://api.ci-ln-k0gim2b-76ef8.aws-2.ci.openshift.org:6443
        name: ci-ln-k0gim2b-76ef8
      contexts:
      - context:
          cluster: ci-ln-k0gim2b-76ef8
          user:
        name:
      current-context:
      kind: Config
      preferences: {}

      Copy to Clipboard Toggle word wrap

      1

      The certificate-authority-data section is not present in externally generated kubeconfig files. It is added with the oc config set command used previously.

   5. Verify the subject and issuer of your customized API server certificate authority by running the following command:

      $ curl --cacert /tmp/caCert.pem https://${fqdn_name}:6443/healthz -v

      Copy to Clipboard Toggle word wrap

      Example output

      Server certificate:
        subject: CN=kas-test-cert_server
        start date: Mar 12 11:39:46 2024 GMT
        expire date: Mar 12 11:39:46 2025 GMT
        subjectAltName: host "dhcp-1-235-3.arm.eng.rdu2.redhat.com" matched cert's "dhcp-1-235-3.arm.eng.rdu2.redhat.com"
        issuer: CN=kas-test-cert_ca
        SSL certificate verify ok.

      Copy to Clipboard Toggle word wrap

      Important

      Either replace the certificate-authority-data in the generated kubeconfig file with the new rootCA or add the certificate-authority-data to the trust root of the operating system. Do not use both methods.

   6. Configure additional CAs in the trust root of the operating system. For example, in the RHEL Client truststore on the client system. See The system-wide truststore for details.

      • Updating the certificate bundle with the configuration that contains the CA is recommended.
      • If you do not want to configure your certificate bundles, you can alternately use the oc login localhost:8443 --certificate-authority=/path/to/cert.crt command, but this method is not preferred.

Procedure

1. From MicroShift, ensure that the certificate is served by the kube-apiserver and verify that the certificate path is appended to the --tls-sni-cert-key FLAG by running the following command:

   $ journalctl -u microshift -b0 | grep tls-sni-cert-key

   Copy to Clipboard Toggle word wrap

   Example output

   Jan 24 14:53:00 localhost.localdomain microshift[45313]: kube-apiserver I0124 14:53:00.649099   45313 flags.go:64] FLAG: --tls-sni-cert-key="[/home/eslutsky/dev/certs/server.crt,/home/eslutsky/dev/certs/server.key;/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.key;/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.key;/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.key

   Copy to Clipboard Toggle word wrap

2. From the client, ensure that the kube-apiserver is serving the correct certificate by running the following command:

   $ openssl s_client -connect <SNI_ADDRESS>:6443 -showcerts | openssl x509 -text -noout -in - | grep -C 1 "Alternative\|CN"

   Copy to Clipboard Toggle word wrap

Procedure

1. Stop the MicroShift services and clean up the custom certificates by running the following command:

   $ sudo microshift-cleanup-data --cert

   Copy to Clipboard Toggle word wrap

   Example output

   Stopping MicroShift services
   Removing MicroShift certificates
   MicroShift service was stopped
   Cleanup succeeded

   Copy to Clipboard Toggle word wrap

2. Restart the MicroShift services to recreate the custom certificates by running the following command:

   $ sudo systemctl start microshift

   Copy to Clipboard Toggle word wrap

Procedure

1. Make a copy of the provided config.yaml.default file in the /etc/microshift/ directory, renaming it config.yaml. Keep the new MicroShift config.yaml you create in the /etc/microshift/ directory. The new config.yaml is read whenever the MicroShift service starts. After you create it, the config.yaml file takes precedence over built-in settings.

2. Replace the default values in the auditLog section of the YAML with your desired valid values.

   Example default auditLog configuration

   apiServer:
   # ....
     auditLog:
       maxFileAge: 7 

   1

   
       maxFileSize: 200 

   2

   
       maxFiles: 1 

   3

   
       profile: Default 

   4

   
   # ....

   Copy to Clipboard Toggle word wrap

   1

   Specifies the maximum time in days that log files are kept. Files older than this limit are deleted. In this example, after a log file is more than 7 days old, it is deleted. The files are deleted regardless of whether or not the live log has reached the maximum file size specified in the maxFileSize field. File age is determined by the timestamp written in the name of the rotated log file, for example, audit-2024-05-16T17-03-59.994.log. When the value is 0, the limit is disabled.

   2

   The maximum audit log file size in megabytes. In this example, the file is rotated as soon as the live log reaches the 200 MB limit. When the value is set to 0, the limit is disabled.

   3

   The maximum number of rotated audit log files retained. After the limit is reached, the log files are deleted in order from oldest to newest. In this example, the value 1 results in only 1 file of size maxFileSize being retained in addition to the current active log. When the value is set to 0, the limit is disabled.

   4

   Logs only metadata for read and write requests; does not log request bodies except for OAuth access token requests. If you do not specify this field, the Default profile is used.

3. Optional: To specify a new directory for logs, you can stop MicroShift, and then move the /var/log/kube-apiserver directory to your desired location:

   1. Stop MicroShift by running the following command:

      $ sudo systemctl stop microshift

      Copy to Clipboard Toggle word wrap

   2. Move the /var/log/kube-apiserver directory to your desired location by running the following command:

      $ sudo mv /var/log/kube-apiserver <~/kube-apiserver> 

      1

      Copy to Clipboard Toggle word wrap

      1

      Replace <~/kube-apiserver> with the path to the directory that you want to use.

   3. If you specified a new directory for logs, create a symlink to your custom directory at /var/log/kube-apiserver by running the following command:

      $ sudo ln -s <~/kube-apiserver> /var/log/kube-apiserver 

      1

      Copy to Clipboard Toggle word wrap

      1

      Replace <~/kube-apiserver> with the path to the directory that you want to use. This enables the collection of logs in sos reports.

4. If you are configuring audit log policies on a running instance, restart MicroShift by entering the following command:

   $ sudo systemctl restart microshift

   Copy to Clipboard Toggle word wrap

Procedure

1. Disable installation of the CSI snapshot controller by entering the optionalCsiComponents value under the storage section of the MicroShift configuration file in /etc/microshift/config.yaml:

   # ...
     storage: {} 

   1

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Accepted values are:

   • Not defining optionalCsiComponents.
   • Specifying optionalCsiComponents field with an empty value ([]) or a single empty string element ([""]).

   • Specifying optionalCsiComponents with one of the accepted values which are snapshot-controller, or none. A value of none is mutually exclusive with all other values.

     Note

     If the optionalCsiComponents value is empty or null, MicroShift defaults to deploying snapshot-controller.

2. After the optionalCsiComponents field is specified with a supported value in the config.yaml, start MicroShift by running the following command:

   $ sudo systemctl start microshift

   Copy to Clipboard Toggle word wrap
   Note

   MicroShift does not redeploy the disabled components after a restart.

Procedure

1. Disable installation of the CSI driver by entering the driver value under the storage section of the MicroShift configuration file in /etc/microshift/config.yaml:

   # ...
     storage
      driver:
      - "none" 

   1

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Valid values are none or lvms.

   Note

   By default, the driver value is empty or null and LVMS is deployed.

2. Start MicroShift after the driver field is specified with a supported value in the /etc/microshift/config.yaml file by running the following command:

   $ sudo systemctl enable --now microshift

   Copy to Clipboard Toggle word wrap
   Note

   MicroShift does not redeploy the disabled components after a restart operation.