Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Configuring

Red Hat build of MicroShift 4.19

Configuring MicroShift

Red Hat OpenShift Documentation Team

Abstract

This document provides instructions for configuring MicroShift.

The MicroShift built-in default settings are listed in a YAML file.

1.1. Configuring Red Hat Device Edge
Copy link

MicroShift and Red Hat Enterprise Linux (RHEL) work together to bring a lighter-weight, single-node Kubernetes to the edge. This combination means that there is a single node that is both control-plane and worker. It also means that the operating system handles many functions. You add features by installing optional RPMs or Operators. In many cases, you must configure the operating system or other resources in addition to the MicroShift service.

Bringing many of these pieces together is the MicroShift configuration file, config.yaml. The MicroShift configuration file customizes your application platform and can enable many advanced functions. For example:

  • Ingress is available by default, but you can add advanced functions such as TLS and route admission specifications by using parameters in the MicroShift configuration file.
  • If you do not need storage, you can disable the built-in storage provider by using the MicroShift configuration file. If you do want to use the built-in storage provider, you must make your adjustments in the lvmd.config file. The role of the MicroShift configuration file in this case is to set whether you use the default storage provider.
  • Advanced networking functions, such as using multiple networks. The Multus package is an installable RPM, but you set up access by using the MicroShift configuration file to set parameters. In addition, you must configure network settings on your networks through the host.

For your convenience, a config.yaml.default file is automatically installed. You can copy and rename this file config.yaml and use it as a starting point for your own custom configuration.

Note

You can also add features that operate without configurations to the MicroShift config.yaml file. For example, you can install and configure GitOps for application management without configuring MicroShift.

Note

If you want to make configuration changes or deploy applications through the MicroShift API with tools other than kustomize manifests, you must wait until the greenboot health checks have finished. This ensures that your changes are not lost if greenboot rolls your rpm-ostree system back to an earlier state.

1.2. The MicroShift configuration file
Copy link

At startup, MicroShift checks the system-wide /etc/microshift/ directory for a configuration file named config.yaml. If the configuration file does not exist in the directory, built-in default values are used to start the service.

You must use the MicroShift configuration file in combination with host and, sometimes, application and service settings. Ensure that you configure each function in tandem when you adjust settings for your MicroShift node.

For your convenience, a config.yaml.default file ready for your inputs is automatically installed.

1.2.1. Default settings
Copy link

If you do not create a config.yaml file or use a configuration snippet YAML file, default values are used. The following example shows the default configuration settings.

  • To see the default values, run the following command: 

    $ microshift show-config
    Copy to Clipboard Toggle word wrap

    Default values example output in YAML form

    apiServer:
  advertiseAddress: 10.44.0.0/32
    1 
    
  auditLog:
    maxFileAge: 0
    maxFileSize: 200
    maxFiles: 10
    profile: Default
  namedCertificates:
    - certPath: ""
      keyPath: ""
      names:
        - ""
  subjectAltNames: []
  tls:
    cipherSuites:
    - ""
    minVersion: VersionTLS12
debugging:
  logLevel: "Normal"
dns:
  baseDomain: microshift.example.com
etcd:
  memoryLimitMB: 0
ingress:
  certificateSecret: router-certs-default
  clientTLS:
    allowedSubjectPatterns:
    clientCA:
      name: ""
    clientCertificatePolicy: ""
  defaultHTTPVersion: 1
  forwardedHeaderPolicy: ""
  httpCompression:
    mimeTypes:
      - ""
  httpEmptyRequestsPolicy: Respond
  listenAddress:
    - ""
  logEmptyRequests: Log
  ports:
    http: 80
    https: 443
  routeAdmissionPolicy:
    namespaceOwnership: InterNamespaceAllowed
    wildcardPolicy: WildcardPolicyAllowed
  status: Managed
  tlsSecurityProfile:
    type: Intermediate
  tuningOptions:
      clientFinTimeout: "1s"
      clientTimeout: "30s"
      headerBufferBytes: 0
      headerBufferMaxRewriteBytes: 0
      healthCheckInterval: "5s"
      maxConnections: 0
      serverFinTimeout: "1s"
      serverTimeout: "30s"
      threadCount: 0
      tlsInspectDelay: "5s"
      tunnelTimeout: "1h"
kubelet:
manifests:
  kustomizePaths:
    - /usr/lib/microshift/manifests
    - /usr/lib/microshift/manifests.d/*
    - /etc/microshift/manifests
    - /etc/microshift/manifests.d/*
network:
  clusterNetwork:
    - 10.42.0.0/16
  cniPlugin: ""
  multus:
    status: Disabled
    2 
    
  serviceNetwork:
    - 10.43.0.0/16
  serviceNodePortRange: 30000-32767
node:
  hostnameOverride: ""
  nodeIP: ""
    3 
    
  nodeIPv6: ""
storage:
  driver: ""
    4 
    
  optionalCsiComponents:
    5 
    
    - ""
telemetry:
  endpoint: https://infogw.api.openshift.com
  proxy: ""
  status: Enabled
    Copy to Clipboard Toggle word wrap

    1
    Calculated based on the address of the service network.
    2
    Controls the deployment of the Multus Container Network Interface (CNI).
    3
    The IP address of the default route.
    4
    Default null value deploys Logical Volume Managed Storage (LVMS).
    5
    Default null value deploys snapshot-controller.

Use the MicroShift YAML file to customize your preferences, settings, and parameters.

2.1. Using custom settings
Copy link

To create custom configurations, make a copy of the config.yaml.default file that is given in the /etc/microshift/ directory, renaming it config.yaml. Keep this file in the /etc/microshift/ directory, and then you can change supported settings that override the defaults before starting or restarting MicroShift.

If you have just a few changes to make to the default settings, consider using configuration drop-in snippets as an alternative method.

Important

Restart MicroShift after changing any configuration settings to have them take effect. The config.yaml file is read only when MicroShift starts.

2.1.1. Separate restarts
Copy link

Applications and other optional services used with your MicroShift node might also need to be restarted separately to apply configuration changes throughout the node. For example, when making changes to certain networking settings, you must stop and restart service and application pods to apply those changes. See each procedure for the task you are completing for more information.

Tip

If you add all of the configurations you need at the same time, you can minimize system restarts.

The following table explains MicroShift configuration YAML parameters and valid values for each:

Expand
Table 2.1. MicroShift config.yaml parameters
FieldTypeDescription

advertiseAddress

string

A string that specifies the IP address from which the API server is advertised to members of the node. The default value is calculated based on the address of the service network.

auditLog.maxFileAge

number

How long log files are stored before automatic deletion. The default value of 0 in the maxFileAge parameter means a log file is never deleted based on age. You can configure this value.

auditLog.maxFileSize

number

By default, when the audit.log file reaches the maxFileSize limit, the audit.log file is rotated and MicroShift begins writing to a new audit.log file. You can configure this value.

auditLog.maxFiles

number

The total number of log files kept. By default, MicroShift retains 10 log files. The oldest is deleted when an excess file is created. You can configure this value.

auditLog.profile

Default, WriteRequestBodies, AllRequestBodies, or None

Logs only metadata for read and write requests; does not log request bodies except for OAuth access token requests. If you do not specify this field, the Default profile is used.

namedCertificates

list

Defines externally generated certificates and domain names by using custom certificate authorities.

namedCertificates.certPath

path

The full path to the certificate.

namedCertificates.keyPath

path

The full path to the certificate key.

namedCertificates.names

list

Optional. Add a list of explicit DNS names. Leading wildcards are allowed. If you do not list names, the implicit names are extracted from the certificates.

subjectAltNames

Fully qualified domain names (FQDNs), wildcards such as *.domain.com, or IP addresses.

Subject Alternative Names for API server certificates. SANs indicate all of the domain names and IP addresses that are secured by a certificate.

tls

list

Defines the transport later protocol (TLS) used and the cipher suites allowed. Provides security for the exposed MicroShift API server and internal control plane endpoints.

tls.cipherSuites

string

Lists the allowed cipher suites that the API server accepts and serves. Defaults to the cipher suites allowed with the TLS specification set in the tls.minVersion parameter.

tls.minVersion

VersionTLS12 or VersionTLS13

Specifies the minimum version of TLS to serve from the API server. The default value is VersionTLS12.

debugging.logLevel

Normal, Debug, Trace, or TraceAll

Log verbosity. The default value is Normal.

dns.baseDomain

valid domain

Base domain of the node. All managed DNS records are subdomains of this base.

etcd.memoryLimitMB

number

By default, etcd uses as much memory as needed to handle the load on the system. However, in memory constrained systems, it might be preferred or necessary to limit the amount of memory etcd can to use at a given time.

ingress.certificateSecret

string

A reference to a secret that contains the default certificate that is served by the ingress controller. When routes do not specify their own certificate, certificateSecret is used.

The secret must contain the following keys and data:

  • tls.crt: certificate file contents
  • tls.key: key file contents

If you do not set one of these values, a wildcard certificate is automatically generated and used. The certificate is valid for the ingress controller domain and subdomains fields, and the generated CA for the certificate is automatically integrated with the truststore for the node.

Any certificate in use is automatically integrated in the MicroShift OAuth server.

ingress.clientTLS

AllowedSubjectPatterns, spec.clientTLS.ClientCA, spec.clientTLS.clientCertificatePolicy

Authenticates client access to the node and services. Mutual TLS authentication is enabled when using these settings. If you do not set values for the spec.clientTLS.clientCertificatePolicy and spec.clientTLS.ClientCA required subfields, client TLS is not enabled.

ingress.clientTLS.AllowedSubjectPatterns

list in PCRE syntax

Optional subfield which specifies a list of regular expressions that are matched against the distinguished name on a valid client certificate to filter requests. Use this parameter to cause the ingress controller to reject certificates based on the distinguished name. The Perl Compatible Regular Expressions (PCRE) syntax is required. If you configure this field, it must contain a valid expression or the MicroShift service fails. At least one pattern must match a client certificate’s distinguished name; otherwise, the ingress controller rejects the certificate and denies the connection.

ingress.clientTLS.ClientCA

string

Required subfield that specifies a config map in the openshift-ingress namespace. The config map must contain a CA certificate bundle.

ingress.clientTLS.ClientCertificatePolicy

Required, Optional

Required subfield that creates a secure route using reencrypt TLS termination with a custom certificate. You must have a certificate/key pair in PEM-encoded files, where the certificate is valid for the route host. The ingress controller only checks client certificates for edge-terminated and reencrypt TLS routes. Certificates for plain text HTTP or passthrough TLS routes are not checked with this setting.

ingress.defaultHTTPVersion

number

Determines the default HTTP version to be used for ingress. The default value is 1, which is the HTTP/1.1 protocol.

ingress.forwardedHeaderPolicy

Append, Replace, IfNone, Never

Specifies when and how the ingress controller sets the Forwarded, X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Port, X-Forwarded-Proto, and X-Forwarded-Proto-Version HTTP headers. The default value is Append.

  • Append specifies that the ingress controller appends existing headers.
  • Replace specifies that the ingress controller sets the headers and replaces any existing Forwarded or X-Forwarded-* headers.
  • IfNone specifies that the ingress controller sets headers if they are not already set.
  • Never specifies that ingress controller never sets the headers, preserving any existing headers.

ingress.httpCompression

object

Defines a policy for HTTP traffic compression. There is no HTTP compression by default.

ingress.httpCompression.mimeTypes

array or null

A list of MIME types to compress. When the list is empty, the ingress controller does not apply any compression. To define a list, use the format of the Content-Type definition in RFC 1341 that specifies the type and subtype of data in the body of a message and the native encoding of the data. For example, Content-Type := type \"/\" subtype *[\";\" parameter].

  • The value of Content-Type can be one of the following types: application, audio, image, message, multipart, text, video, or a custom type preceded by \"X-\" and followed by a token. The token must be defined in one of the following ways:
  • The token is a string of at least one character, and does not contain white spaces, control characters, or any of the characters in the tspecials set.
  • The tspecials set contains the characters ()\u003c\u003e@,;:\\\"/[]?.=.
  • The subtype in Content-Type is also a token.
  • The optional parameters following the subtype are defined as token \"=\" (token / quoted-string).
  • The quoted-string, as defined in RFC 822, is surrounded by double quotes and can contain white spaces plus any character except \\, \", and CR. The quoted-string can also contain any single ASCII character if it is escaped by the following characters: \\.",.

Not all MIME types benefit from compression, but HAProxy uses resources to try to compress files when compression is configured. Generally speaking, text formats such as html, ccs, and js benefit from compression. Spending CPU resources to compress file types that are already compressed, such as images, audio, and video, is probably not worth the limited benefit.

ingress.httpEmptyRequestsPolicy

Respond or Ignore

The default value is Respond. Describes how HTTP connections should be handled if the connection times out before a request is received. These connections typically come from the health probes of a load balancer service or a web browser’s speculative connections, such as a preconnect.

  • If the field is set to Respond, the ingress controller sends an "HTTP 400" or "408" response, logs the connection if access logging is enabled, and counts the connection in the appropriate metrics.
  • If the field is set to Ignore, the ingress controller closes the connection without sending a response, logging the connection, or incrementing metrics. Setting this field to Ignore might impede detection and diagnosis of problems or intrusions, especially when timed-out connections are caused by network errors or port scans. In both cases, logging empty requests can be useful for diagnosing errors and detecting intrusion attempts.

ingress.listenAddress

IP address, NIC name, or multiple

Value defaults to the entire network of the host. The valid configurable value is a list that can be either a single IP address or NIC name or multiple IP addresses and NIC names.

ingress.logEmptyRequests

Log or Ignore

The default value is Log. Specifies how connections on which empty requests are received are logged. These connections typically come from the health probes of a load balancer service health or a web browser’s speculative connections, such as a preconnect. Logging typical requests might be undesirable, but requests can also be caused by network errors or port scans, in which case logging can be useful for diagnosing errors and detecting intrusion attempts.

ingress.ports.http

80

The default port shown. Configurable. Valid value is a single, unique port in the 1-65535 range. The values of the ports.http and ports.https fields cannot be the same.

ingress.ports.https

443

The default port shown. Configurable. Valid value is a single, unique port in the 1-65535 range. The values of the ports.http and ports.https fields cannot be the same.

ingress.routeAdmissionPolicy

namespaceOwnership or wildcardPolicy

Defines a policy for handling new route claims, such as allowing or denying claims across namespaces. By default, allows routes to claim different paths of the same hostname across namespaces.

ingress.routeAdmissionPolicy.namespaceOwnership

Strict or InterNamespaceAllowed

Describes how hostname claims across namespaces should be handled. The default value is InterNamespaceAllowed. Specifying Strict prevents routes in different namespaces from claiming the same hostname. If the value is deleted in a customized MicroShift config.yaml file, the InterNamespaceAllowed value is automatically set.

  • Strict: does not allow routes to claim the same hostname across namespaces.
  • InterNamespaceAllowed: allows routes to claim different paths of the same hostname across namespaces.

ingress.routeAdmissionPolicy.wildcardPolicy

WildcardsAllowed or WildcardsDisallowed

Describes how routes with wildcard policies are handled by the ingress controller.

  • WildcardsAllowed: Indicates routes with any wildcard policy are admitted by the ingress controller.
  • WildcardsDisallowed: Indicates only routes with a wildcard policy of None are admitted by the ingress controller. Updating wildcardPolicy from WildcardsAllowed to WildcardsDisallowed causes admitted routes with a wildcard policy of Subdomain to stop working. These routes must be recreated to a wildcard policy of None to be readmitted by the ingress controller. WildcardsDisallowed is the default setting.

ingress.status

Managed or Removed

Router status. The default is Managed.

ingress.tlsSecurityProfile

object

Specifies settings for ingress controllers TLS connections. If you do not set one, the default value is based on the apiservers.config.openshift.io/cluster resource.

ingress.tlsSecurityProfile.type

Old, Intermediate, Modern, Custom

Specifies the profile type for the TLS Security. The default value is Intermediate.

When using the Old, Intermediate, and Modern profile types, the effective profile configuration is subject to change between releases. For example, given a specification to use the Intermediate profile deployed on release X.Y.Z, an upgrade to release X.Y.Z+1 might cause a new profile configuration to be applied to the ingress controller, resulting in a rollout.

ingress.tlsSecurityProfile.minTLSVersion

number

Specifies the TLS version for ingress controllers.

The minimum TLS version is 1.1, and the maximum TLS version is 1.3.

  • Ciphers and the minimum TLS version of the configured security profile are reflected in the TLSProfile status.
  • The ingress controller converts the TLS 1.0 of an Old or Custom profile to 1.1.

ingress.tuningOptions

Objects

Specifies options for tuning the performance of ingress controller pods.

ingress.tuningOptions.clientFinTimeout

string with format duration

Defines how long a connection is held open while waiting for a client response to the server/backend before closing the connection. The default timeout is 1s, which is 1 second.

ingress.tuningOptions.clientTimeout

string with format duration

Defines how long a connection is held open while waiting for a client response. The default timeout is 30s, which is 30 seconds.

ingress.tuningOptions.headerBufferBytes

An integer with the format of int32; 16384 is the minimum value when HTTP/2 is enabled.

Describes how much memory in bytes must be reserved for IngressController connection sessions. The default value is 32768 in bytes.

  • Setting this field is generally not recommended because headerBufferBytes values that are too small can break the IngressController and headerBufferBytes values that are too large can cause the IngressController to use significantly more memory than necessary.

ingress.tuningOptions.headerBufferMaxRewriteBytes

integer, formatted int32; 4096 is the minimum value

Describes how much memory in bytes must be reserved from headerBufferBytes for HTTP header rewriting and appending for IngressController connection sessions. The default value is 8192 bytes. Incoming HTTP requests are limited to the headerBufferBytes bytes minus the headerBufferMaxRewriteBytes bytes, meaning that the value of headerBufferBytes must be greater than the value of headerBufferMaxRewriteBytes.

  • Setting this field is generally not recommended because headerBufferMaxRewriteBytes values that are too small can break the IngressController and headerBufferMaxRewriteBytes values that are too large can cause the IngressController to use significantly more memory than necessary.

ingress.tuningOptions.healthCheckInterval: ""

string with pattern: ^(0|([0-9]+(\\.[0-9]+)?(ns|us|µs|μs|ms|s|m|h))+)$

The default healthCheckInterval value is 5s, which is 5 seconds. This parameter value defines how long the router waits between two consecutive health checks on the router’s configured backends. The minimum allowed value is 1s and the maximum allowed value is 2147483647ms, which is 24.85 days.

  • This value is applied globally as a default for all routes, but can be overridden per-route by the route annotation router.openshift.io/haproxy.health.check.interval.
  • Requires an unsigned duration string of decimal numbers, each with an optional fraction and unit suffix, such as 300ms, 1.5h or 2h45m. Valid time units are ns, us (or µs U+00B5 or μs U+03BC), ms, s, m, h.
  • Setting this parameter value to less than 5s can cause excess traffic due to too frequent TCP health checks and accompanying SYN packet storms.
  • Setting this parameter value too high can result in increased latency because of backend servers that are no longer available, but have not yet been detected as such.
  • An empty or 0 value means "no opinion" and the ingress controller chooses a default. Note that the default value might change in future releases.

ingress.tuningOptions.maxConnections

integer, valid values are: empty, 0, -1, and the range 2000-2000000

The default value is 0. defines the maximum number of simultaneous connections that can be established per HAProxy process. Increasing this value allows each ingress controller pod to handle more connections at the cost of additional system resources being consumed.

  • If this field is empty or 0, the IngressController uses the default value of 50000, but the default is subject to change in future releases.
  • If the value is -1, then HAProxy dynamically computes a maximum value based on the available resources set with ulimit values in the running container. Selecting -1, which means auto, results in a large value being computed, and therefore each HAProxy process incurs significant memory usage compared with the current default of 50000.
  • Setting a value that is greater than the current operating system limit prevents the HAProxy process from starting.

  • You can monitor memory usage for router containers with the following metric: 

    container_memory_working_set_bytes{container=`router`,namespace=`openshift-ingress`}`
    Copy to Clipboard Toggle word wrap

  • You can monitor memory usage of individual `HAProxy`processes in router containers with the following metric: 

    container_memory_working_set_bytes{container=`router`,namespace=`openshift-ingress`}/container_processes{container=`router`,namespace=`openshift-ingress`}
    Copy to Clipboard Toggle word wrap

ingress.tuningOptions.serverFinTimeout

string in the format duration

Defines how long a connection is held open while waiting for a server or backend response to the client before closing the connection. The default timeout is 1s.

ingress.tuningOptions.serverTimeout

string in the format duration

Defines how long a connection is held open while waiting for a server or backend response. The default timeout is 30s.

ingress.tuningOptions.threadCount

integer in the form int32; minimum value is 1, maximum is 64

Defines the number of threads created per HAProxy process. The default value is 4. If this field is empty, the default value is used.

  • Setting this field is generally not recommended. Creating more threads allows each ingress controller pod to handle more connections at the cost of more system resources being used. Increasing the number of HAProxy threads allows the ingress controller pods to use more CPU time under load, potentially starving other pods if set too high. Conversely, reducing the number of threads may cause the ingress controller to perform poorly.

ingress.tuningOptions.tlsInspectDelay

string in the format duration

Defines how long the router can hold data to find a matching route. Setting this interval with too short a value can cause the router to revert to the default certificate for edge-terminated clients or re-encrypt routes, even when a better-matching certificate could be used.

  • The default inspect delay is 5s which is 5 seconds, which is expected to be sufficient for most cases. Increasing the value of this configuration specifically for high-latency networks can cause a delay in finishing the SSL handshake. Any configured value must be transparent to your application.

ingress.tuningOptions.tunnelTimeout

string in the format duration

Defines how long a tunnel connection, including websockets, are held open while the tunnel is idle. The default timeout is 1h, which is 1 hour.

kubelet

See the MicroShift low-latency instructions

Parameter for passthrough configuration of the kubelet node agent. Used for low-latency configuration. The default value is null.

manifests

list of paths

The locations on the file system to scan for kustomization files to use to load manifests. Set to a list of paths to scan only those paths. Set to an empty list to disable loading manifests. The entries in the list can be glob patterns to match multiple subdirectories. The default values are /usr/lib/microshift/manifests, /usr/lib/microshift/manifests.d/, /etc/microshift/manifests, and /etc/microshift/manifests.d/.

network.clusterNetwork

IP address block

A block of IP addresses from which pod IP addresses are allocated. IPv4 is the default network. Dual-stack entries are supported. The first entry in this field is immutable after MicroShift starts. The default range is 10.42.0.0/16.

network.cniPlugin

String

Deploys the Open Virtual Networking - Kubernetes (OVN-K) network plugin as the default container network interface (CNI) when empty or set to "ovnk". Supported values are empty, "" or "ovnk". Setting to "none" removes the CNI and is not recommended. Only OVN-K is managed by MicroShift.

network.multus.status

string

Controls the deployment of the Multus Container Network Interface (CNI). The default status is Disabled. If you set the value to Enabled, the Multus CNI cannot be deleted.

network.serviceNetwork

IP address block

A block of virtual IP addresses for Kubernetes services. IP address pool for services. IPv4 is the default. Dual-stack entries are supported. The first entry in this field is immutable after MicroShift starts. The default range is 10.43.0.0/16.

network.serviceNodePortRange

range

The port range allowed for Kubernetes services of type NodePort. If you do not specify the range, the default range of 30000-32767 is used. Services without a NodePort specified are automatically allocated one from this range. This parameter can be updated after MicroShift starts.

node.hostnameOverride

string

The name of the node. The default value is the hostname. If non-empty, this string is used to identify the node instead of the hostname. This value is immutable after MicroShift starts.

node.nodeIP

IPv4 address

The IPv4 address of the node. The default value is the IP address of the default route.

nodeIPv6

IPv6 address

The IPv6 address for the node for dual-stack configurations. Cannot be configured in single stack for either IPv4 or IPv6. The default is an empty value or null.

storage.driver

none or lvms

The default value is empty. An empty value or null field defaults to LVMS deployment.

storage.optionalCsiComponents

array

The default value is null or an empty array. A null or empty array defaults to deploying snapshot-controller. Expected values are csi-snapshot-controller or none. A value of none is mutually exclusive with all other values.

telemetry.endpoint

https://infogw.api.openshift.com

The endpoint where telemetry data is sent. No user or private data is included in the metrics reported. The default value is https://infogw.api.openshift.com.

telemetry.status

Enabled

Telemetry status, which can be Enabled or Disabled. The default value is Enabled.

The apiserver.advertiseAddress flag specifies the IP address on which to advertise the API server to members of the node. This address must be reachable by the node. You can set a custom IP address here, but you must also add the IP address to a host interface. Customizing this parameter preempts MicroShift from adding a default IP address to the br-ex network interface.

Important

If you customize the advertiseAddress IP address, make sure it is reachable by the node when MicroShift starts by adding the IP address to a host interface.

If unset, the default value is set to the next immediate subnet after the service network. For example, when the service network is 10.43.0.0/16, the advertiseAddress is set to 10.44.0.0/32.

The serviceNodePortRange setting extends the port range available to NodePort services. This option is useful when specific standard ports under the 30000-32767 range need to be exposed. For example, if your device needs to expose the 1883/tcp MQ Telemetry Transport (MQTT) port on the network because client devices cannot use a different port.

Important

NodePorts can overlap with system ports, causing a malfunction of the system or MicroShift.

Consider the following when configuring the NodePort service ranges:

  • Do not create any NodePort service without an explicit nodePort selection. When an explicit nodePort is not specified, the port is assigned randomly by the kube-apiserver and cannot be predicted.
  • Do not create any NodePort service for any system service port, MicroShift port, or other services you expose on your device HostNetwork.

  • Table one specifies ports to avoid when extending the port range:

    Expand
    Table 2.2. Ports to avoid.
    PortDescription

    22/tcp

    SSH port

    80/tcp

    OpenShift Router HTTP endpoint

    443/tcp

    OpenShift Router HTTPS endpoint

    1936/tcp

    Metrics service for the openshift-router, not exposed today

    2379/tcp

    etcd port

    2380/tcp

    etcd port

    6443

    Kubernetes API

    8445/tcp

    openshift-route-controller-manager

    9537/tcp

    cri-o metrics

    10250/tcp

    kubelet

    10248/tcp

    kubelet healthz port

    10259/tcp

    kube scheduler

Chapter 3. Using configuration snippets
Copy link

If you want to configure one or two settings, use the /etc/microshift/config.d/ configuration directory to drop in configuration snippet YAML files.

3.1. How configuration snippets work
Copy link

If you want to configure one or two settings, such as adding subject alternative names (SANs), you can use the /etc/microshift/config.d/ configuration directory to drop in configuration snippet YAML files. You must restart MicroShift for new configurations to apply.

To return to previous values, you can delete a configuration snippet and restart MicroShift.

At runtime, the YAML files inside /etc/microshift/config.d are merged into the existing MicroShift configuration, whether that configuration is a result of default values or a user-created config.yaml file. You do not need to create a config.yaml file to use a configuration snippet.

Files in the snippet directory are sorted in lexicographical order and run sequentially. You can use numerical prefixes for snippets so that each is read in the order you want. The last-read file takes precedence when there is more than one YAML for the same parameter.

Important

Configuration snippets take precedence over both default values and a customized config.yaml configuration file.

Lists, or arrays, are not merged, they are overwritten. For example, you can replace a SAN or list of SANs by creating an additional snippet for the same field that is read after the first:

MicroShift configuration directory contents

  • /etc/microshift/config.yaml.default or /etc/microshift/config.yaml

Example MicroShift configuration snippet directory contents

  • /etc/microshift/config.d/10-san.yaml

  • /etc/microshift/config.d/20-san.yaml

    Example 10-san.yaml snippet

    apiServer:
  subjectAltNames:
    - host1
    - host2
    Copy to Clipboard Toggle word wrap

    Example 20-san.yaml snippet

    apiServer:
  subjectAltNames:
    - hostZ
    Copy to Clipboard Toggle word wrap

    Example configuration result

    apiServer:
  subjectAltNames:
    - hostZ
    Copy to Clipboard Toggle word wrap

If you want to add a value to an existing list, you can add it to an existing snippet. For example, to add hostZ to an existing list of SANs, edit the snippet you have instead of creating a new one:

Example 10-san.yaml snippet

apiServer:
  subjectAltNames:
    - host1
    - host2
    - hostZ
Copy to Clipboard Toggle word wrap

Example configuration result

apiServer:
  subjectAltNames:
    - host1
    - host2
    - hostZ
Copy to Clipboard Toggle word wrap

Objects are merged together when you use a configuration snippet.

Example 10-advertiseAddress.yaml snippet

apiServer:
  advertiseAddress: "microshift-example"
Copy to Clipboard Toggle word wrap

Example 20-audit-log.yaml snippet

apiServer:
  auditLog:
    maxFileAge: 12
Copy to Clipboard Toggle word wrap

Example configuration result

apiServer:
  advertiseAddress: "microshift-example"
  auditLog:
    maxFileAge: 12
Copy to Clipboard Toggle word wrap

3.4. Examples of mixed configuration snippets
Copy link

In this example, the values of both advertiseAddress and auditLog.maxFileAge fields merge into the configuration, but only the c.com and d.com subjectAltNames values are retained. This happens because the numbering in the filename indicates that the c.com and d.com values are higher priority.

Example 10-advertiseAddress.yaml snippet

apiServer:
  advertiseAddress: "microshift-example"
Copy to Clipboard Toggle word wrap

Example 20-audit-log.yaml snippet

apiServer:
  auditLog:
    maxFileAge: 12
Copy to Clipboard Toggle word wrap

Example 30-SAN.yaml snippet

apiServer:
  subjectAltNames:
    - a.com
    - b.com
Copy to Clipboard Toggle word wrap

Example 40-SAN.yaml snippet

apiServer:
  subjectAltNames:
    - c.com
    - d.com
Copy to Clipboard Toggle word wrap

Example configuration result

apiServer:
  advertiseAddress: "microshift-example"
  auditLog:
    maxFileAge: 12
  subjectAltNames:
    - c.com
    - d.com
Copy to Clipboard Toggle word wrap

You can use the IPv6 networking protocol in either single-stack or dual-stack networking modes.

4.1. IPv6 networking with MicroShift
Copy link

The MicroShift service defaults to IPv4 address families node-wide. However, IPv6 single-stack and IPv4/IPv6 dual-stack networking is available on supported platforms.

  • When you set the values for IPv6 in the MicroShift configuration file and restart the service, settings managed by the OVN-Kubernetes network plugin are updated automatically.
  • After migrating to dual-stack networking, both new and existing pods have dual-stack networking enabled.
  • If you require node-wide IPv6 access, such as for the control plane and other services, use the following configuration examples. The MicroShift Multus Container Network Interface (CNI) plugin can enable IPv6 for pods.
  • For dual-stack networking, each MicroShift node network and service network supports up to two values in the node and service network configuration parameters.
Important

Plan for IPv6 before starting MicroShift for the first time. Switching a node to and from different IP families is not supported unless you are migrating a node from default single-stack to dual-stack networking.

If you configure your networking for either IPv6 single stack or IPv4/IPv6 dual stack, you must restart application pods and services. Otherwise pods and services remain configured with the default IP family.

4.2. Configuring IPv6 single-stack networking
Copy link

You can use the IPv6 network protocol by updating the MicroShift service configuration file.

Prerequisites

  • You installed the OpenShift CLI (oc).
  • You have root access to the node.
  • Your node uses the OVN-Kubernetes network plugin.
  • The host has an IPv6 address and IPv6 routes, including the default.

Procedure

  1. If you have not done so, make a copy of the provided config.yaml.default file in the /etc/microshift/ directory, renaming it config.yaml.

  2. Keep the new MicroShift config.yaml in the /etc/microshift/ directory. Your config.yaml file is read every time the MicroShift service starts.

    Note

    After you create it, the config.yaml file takes precedence over built-in settings.

  3. Replace the default values in the network section of the MicroShift YAML with your valid values.

    Example single-stack IPv6 networking configuration

    apiServer:
# ...
network:
  clusterNetwork:
  - fd01::/48
    1 
    
  serviceNetwork:
  - fd02::/112
    2 
    
node:
  nodeIP: 2600:1f14:1c48:ee00:2d76:3190:5bc2:5aef
    3 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    Specify a clusterNetwork with a CIDR value that is less than 64.
    2
    Specify an IPv6 CIDR with a prefix of 112. Kubernetes uses only the lowest 16 bits. For a prefix of 112, IP addresses are assigned from 112 to 128 bits.
    3
    Example node IP address. Valid values are IP addresses in the IPv6 address family. You must only specify an IPv6 address when an IPv4 network is also present. If an IPv4 network is not present, the MicroShift service automatically fills in this value upon restart.

  4. Complete any other configurations you require, then start MicroShift by running the following command: 

    $ sudo systemctl start microshift
    Copy to Clipboard Toggle word wrap

Verification

  1. Retrieve the networks defined in the node resource by running the following command: 

    $ oc get node -o jsonpath='{.items[].spec.podCIDRs[]}'
    Copy to Clipboard Toggle word wrap

    Example output

    fd01::/48
    Copy to Clipboard Toggle word wrap

  2. Retrieve the status of the pods by running the following command: 

    $ oc get pod -A -o wide
    Copy to Clipboard Toggle word wrap

    Example output

    NAMESPACE                  NAME                                      READY   STATUS    RESTARTS   AGE   IP                      NODE           NOMINATED NODE   READINESS GATES
kube-system                csi-snapshot-controller-bb7cb654b-rqrt6   1/1     Running   0          65s   fd01:0:0:1::5           microshift-9   <none>           <none>
openshift-dns              dns-default-cjn66                         2/2     Running   0          62s   fd01:0:0:1::9           microshift-9   <none>           <none>
openshift-dns              node-resolver-ppnjb                       1/1     Running   0          63s   2001:db9:ca7:ff::1db8   microshift-9   <none>           <none>
openshift-ingress          router-default-6d97d7b8b6-wdtmg           1/1     Running   0          61s   fd01:0:0:1::8           microshift-9   <none>           <none>
openshift-ovn-kubernetes   ovnkube-master-gfvp5                      4/4     Running   0          63s   2001:db9:ca7:ff::1db8   microshift-9   <none>           <none>
openshift-ovn-kubernetes   ovnkube-node-bnpjh                        1/1     Running   0          63s   2001:db9:ca7:ff::1db8   microshift-9   <none>           <none>
openshift-service-ca       service-ca-5d7bd9db6-j25bd                1/1     Running   0          60s   fd01:0:0:1::4           microshift-9   <none>           <none>
openshift-storage          lvms-operator-656cd9b59b-bwr47            1/1     Running   0          63s   fd01:0:0:1::7           microshift-9   <none>           <none>
openshift-storage          vg-manager-f7dmk                          1/1     Running   0          27s   fd01:0:0:1::a           microshift-9   <none>           <none>
    Copy to Clipboard Toggle word wrap

  3. Retrieve the status of services by running the following command: 

    $ oc get svc -A
    Copy to Clipboard Toggle word wrap

    Example output

    NAMESPACE           NAME                            TYPE           CLUSTER-IP   EXTERNAL-IP                                             PORT(S)                      AGE
default             kubernetes                      ClusterIP      fd02::1      <none>                                                  443/TCP                      3m42s
openshift-dns       dns-default                     ClusterIP      fd02::a      <none>                                                  53/UDP,53/TCP,9154/TCP       2m58s
openshift-ingress   router-default                  LoadBalancer   fd02::f2e6   2001:db9:ca7:ff::1db8,fd01:0:0:1::2,fd02::1:0,fd69::2   80:31133/TCP,443:31996/TCP   2m58s
openshift-ingress   router-internal-default         ClusterIP      fd02::c55e   <none>                                                  80/TCP,443/TCP,1936/TCP      2m58s
openshift-storage   lvms-operator-metrics-service   ClusterIP      fd02::7afb   <none>                                                  443/TCP                      2m58s
openshift-storage   lvms-webhook-service            ClusterIP      fd02::d8dd   <none>                                                  443/TCP                      2m58s
openshift-storage   vg-manager-metrics-service      ClusterIP      fd02::fc1    <none>                                                  443/TCP                      2m58s
    Copy to Clipboard Toggle word wrap

You can configure your MicroShift node to run on dual-stack networking that supports IPv4 and IPv6 address families by using the configuration file before starting the service.

  • The first IP family in the configuration is the primary IP stack in the node.
  • After the node is running with dual-stack networking, enable application pods and add-on services for dual-stack by restarting them.
Important

The OVN-Kubernetes network plugin requires that both IPv4 and IPv6 default routes be on the same network device. IPv4 and IPv6 default routes on separate network devices is not supported.

Important

When using dual-stack networking where IPv6 is required, you cannot use IPv4-mapped IPv6 addresses, such as ::FFFF:198.51.100.1.

Prerequisites

  • You installed the OpenShift CLI (oc).
  • You have root access to the node.
  • Your node uses the OVN-Kubernetes network plugin.
  • The host has both IPv4 and IPv6 addresses and routes, including a default for each.
  • The host has at least two L3 networks, IPv4 and IPv6.

Procedure

  1. If you have not done so, make a copy of the provided config.yaml.default file in the /etc/microshift/ directory, renaming it config.yaml.

  2. Keep the new MicroShift config.yaml in the /etc/microshift/ directory. Your config.yaml file is read every time the MicroShift service starts.

    Note

    After you create it, the config.yaml file takes precedence over built-in settings.

  3. If you have not started MicroShift, replace the default values in the network section of the MicroShift YAML with your valid values.

    Example dual-stack IPv6 networking configuration with network assignments

    apiServer:
# ...
apiServer:
  subjectAltNames:
  - 192.168.113.117
  - 2001:db9:ca7:ff::1db8
network:
  clusterNetwork:
  - 10.42.0.0/16
  - fd01::/48
    1 
    
  serviceNetwork:
  - 10.43.0.0/16
  - fd02::/112
    2 
    
node:
  nodeIP: 192.168.113.117
    3 
    
  nodeIPv6: 2001:db9:ca7:ff::1db8
    4 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    Specify an IPv6 clusterNetwork with a CIDR value that is less than 64.
    2
    Specify an IPv6 CIDR with a prefix of 112. Kubernetes uses only the lowest 16 bits. For a prefix of 112, IP addresses are assigned from 112 to 128 bits.
    3
    Example node IP address. Must be an IPv4 address family.
    4
    Example node IP address for dual-stack configuration. Must be an IPv6 address family. Configurable only with dual-stack networking.

  4. Complete any other MicroShift configurations you require, then start MicroShift by running the following command: 

    $ sudo systemctl start microshift
    Copy to Clipboard Toggle word wrap
  5. Reset the IP family policy for application pods and services as needed, then restart those application pods and services to enable dual-stack networking. See "Resetting the IP family policy for application pods and services" for a simple example.

Verification

  1. You can verify that all of the system services and pods to have two IP addresses, one for each family, by using the following steps:

    1. Retrieve the networks defined in the node resource by running the following command: 

      $ oc get pod -n openshift-ingress router-default-5b75594b4-w7w6s -o jsonpath='{.status.podIPs}'
      Copy to Clipboard Toggle word wrap

      Example output

      [{"ip":"10.42.0.4"},{"ip":"fd01:0:0:1::4"}]
      Copy to Clipboard Toggle word wrap

    2. Retrieve the networks defined by the host network pods by running the following command: 

      $ oc get pod -n openshift-ovn-kubernetes ovnkube-master-2fm2k -o jsonpath='{.status.podIPs}'
      Copy to Clipboard Toggle word wrap

      Example output

      [{"ip":"192.168.113.117"},{"ip":"2001:db9:ca7:ff::1db8"}]
      Copy to Clipboard Toggle word wrap

You can convert a single-stack node to dual-stack node networking that supports IPv4 and IPv6 address families by setting two entries in the service and node network parameters in the MicroShift configuration file.

  • The first IP family in the configuration is the primary IP stack in the node.
  • MicroShift system pods and services are automatically updated upon MicroShift restart.
  • After the node is migrated to dual-stack networking and has restarted, enable workload pods and services for dual-stack networking by restarting them.
Important

The OVN-Kubernetes network plugin requires that both IPv4 and IPv6 default routes be on the same network device. IPv4 and IPv6 default routes on separate network devices is not supported.

Important

When using dual-stack networking where IPv6 is required, you cannot use IPv4-mapped IPv6 addresses, such as ::FFFF:198.51.100.1.

Prerequisites

  • You installed the OpenShift CLI (oc).
  • You have root access to the node.
  • Your node uses the OVN-Kubernetes network plugin.
  • The host has both IPv4 and IPv6 addresses and routes, including a default for each.
  • The host has at least two L3 networks, IPv4 and IPv6.

Procedure

  1. If you have not done so, make a copy of the provided config.yaml.default file in the /etc/microshift/ directory, renaming it config.yaml.

  2. Keep the new MicroShift config.yaml in the /etc/microshift/ directory. Your config.yaml file is read every time the MicroShift service starts.

    Note

    After you create it, the config.yaml file takes precedence over built-in settings.

  3. Add IPv6 configurations to the network section of the MicroShift YAML with your valid values:

    Warning

    You must keep the same first entry across restarts and migrations. This is true for any migration: single-to-dual stack, or dual-to-single stack. A complete wipe of the etcd database is required if a change to the first entry is needed. This might result in application data loss and is not supported.

    1. Add an IPv6 configuration for a second network in the network section of the MicroShift YAML with your valid values.

    2. Add network assignments to the network section of the MicroShift config.yaml to enable dual stack with IPv6 as secondary network.

      Example dual-stack IPv6 configuration with network assignments

      # ...
apiServer:
  subjectAltNames:
  - 192.168.113.117
  - 2001:db9:ca7:ff::1db8
      1 
      
network:
  clusterNetwork:
  - 10.42.0.0/16
      2 
      
  - fd01::/48
      3 
      
  serviceNetwork:
  - 10.43.0.0/16
  - fd02::/112
      4 
      
node:
  nodeIP: 192.168.113.117
      5 
      
  nodeIPv6: 2001:db9:ca7:ff::1db8
      6 
      
# ...
      Copy to Clipboard Toggle word wrap

      1
      The IPv6 node address.
      2
      IPv4 network. Specify a clusterNetwork with a CIDR value that is less than 24.
      3
      IPv6 network. Specify a clusterNetwork with a CIDR value that is less than 64.
      4
      Specify an IPv6 CIDR with a prefix of 112. Kubernetes uses only the lowest 16 bits. For a prefix of 112, IP addresses are assigned from 112 to 128 bits.
      5
      Example node IP address. Maintain the previous IPv4 IP address.
      6
      Example node IP address. Must be an IPv6 address family.

  4. Complete any other configurations you require, then restart MicroShift by running the following command: 

    $ sudo systemctl restart microshift
    Copy to Clipboard Toggle word wrap
  5. Reset the IP family policy for application pods and services as needed, then restart those application pods and services to enable dual-stack networking. See "Resetting the IP family policy for application pods and services" for a simple example.

Verification

You can verify that all of the system services and pods to have two IP addresses, one for each family, by using the following steps:

  1. Retrieve the status of the pods by running the following command: 

    $ oc get pod -A -o wide
    Copy to Clipboard Toggle word wrap

    Example output

    NAMESPACE                  NAME                                      READY   STATUS    RESTARTS        AGE     IP                NODE           NOMINATED NODE   READINESS GATES
kube-system                csi-snapshot-controller-bb7cb654b-7s5ql   1/1     Running   0               46m     10.42.0.6         microshift-9   <none>           <none>
openshift-dns              dns-default-zxkqn                         2/2     Running   0               46m     10.42.0.5         microshift-9   <none>           <none>
openshift-dns              node-resolver-r2h5z                       1/1     Running   0               46m     192.168.113.117   microshift-9   <none>           <none>
openshift-ingress          router-default-5b75594b4-228z7            1/1     Running   0               2m5s    10.42.0.3         microshift-9   <none>           <none>
openshift-ovn-kubernetes   ovnkube-master-bltk7                      4/4     Running   2 (2m32s ago)   2m36s   192.168.113.117   microshift-9   <none>           <none>
openshift-ovn-kubernetes   ovnkube-node-9ghgs                        1/1     Running   2 (2m32s ago)   46m     192.168.113.117   microshift-9   <none>           <none>
openshift-service-ca       service-ca-5d7bd9db6-qgwgw                1/1     Running   0               46m     10.42.0.7         microshift-9   <none>           <none>
openshift-storage          lvms-operator-656cd9b59b-8rpf4            1/1     Running   0               46m     10.42.0.8         microshift-9   <none>           <none>
openshift-storage          vg-manager-wqmh4                          1/1     Running   2 (2m39s ago)   46m     10.42.0.10        microshift-9   <none>           <none>
    Copy to Clipboard Toggle word wrap

  2. Retrieve the networks defined by the OVN-K network plugin by running the following command: 

    $ oc get pod -n openshift-ovn-kubernetes ovnkube-master-bltk7 -o jsonpath='{.status.podIPs}'
    Copy to Clipboard Toggle word wrap

    Example output

    [{"ip":"192.168.113.117"},{"ip":"2001:db9:ca7:ff::1db8"}]
    Copy to Clipboard Toggle word wrap

  3. Retrieve the networks defined in the node resource by running the following command: 

    $ oc get pod -n openshift-ingress router-default-5b75594b4-228z7 -o jsonpath='{.status.podIPs}'
    Copy to Clipboard Toggle word wrap

    Example output

    [{"ip":"10.42.0.3"},{"ip":"fd01:0:0:1::3"}]
    Copy to Clipboard Toggle word wrap

    Note

    To return to single-stack networking, you can remove the second entry to the networks and return to the single stack that was configured before migrating to dual-stack.

The default ipFamilyPolicy configuration value, PreferSingleStack, does not automatically update in all services after you update your MicroShift configuration to dual-stack networking. To enable dual-stack networking in services and application pods, you must update the ipFamilyPolicy value.

Prerequisites

  • You used the MicroShift config.yaml to define a dual-stack network with an IPv6 address family.

Procedure

  1. Set the spec.ipFamilyPolicy field to a valid value for dual-stack networking in your service or pod by using the following example:

    Example dual-stack network configuration for a service

    kind: Service
apiVersion: v1
metadata:
  name: microshift-new-service
  labels: app: microshift-application
spec:
  type: NodePort
  ipFamilyPolicy: `PreferDualStack`
    1 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    Required. Valid values for dual-stack networking are PreferDualStack and RequireDualStack. The value you set depends on the requirements of your application. PreferSingleStack is the default value for the ipFamilyPolicy field.
  2. Restart any application pods that do not have a hostNetwork defined. Pods that do have a hostNetwork defined do not need to be restarted to update the ipFamilyPolicy value.
Note

MicroShift system services and pods are automatically updated when the ipFamilyPolicy value is updated.

The OVN-Kubernetes network plugin has the following limitations:

  • For a cluster configured for dual-stack networking, both IPv4 and IPv6 traffic must use the same network interface as the default gateway.

    If this requirement is not met, pods on the host in the ovnkube-node daemon set enter the CrashLoopBackOff state.

    If you display a pod with a command such as oc get pod -n openshift-ovn-kubernetes -l app=ovnkube-node -o yaml, the status field has more than one message about the default gateway, as shown in the following output: 

    I1006 16:09:50.985852   60651 helper_linux.go:73] Found default gateway interface br-ex 192.168.127.1
I1006 16:09:50.985923   60651 helper_linux.go:73] Found default gateway interface ens4 fe80::5054:ff:febe:bcd4
F1006 16:09:50.985939   60651 ovnkube.go:130] multiple gateway interfaces detected: br-ex ens4
    Copy to Clipboard Toggle word wrap

    The only resolution is to reconfigure the host networking so that both IP families use the same network interface for the default gateway.

  • For a cluster configured for dual-stack networking, both the IPv4 and IPv6 routing tables must contain the default gateway.

    If this requirement is not met, pods on the host in the ovnkube-node daemon set enter the CrashLoopBackOff state.

    If you display a pod with a command such as oc get pod -n openshift-ovn-kubernetes -l app=ovnkube-node -o yaml, the status field has more than one message about the default gateway, as shown in the following output: 

    I0512 19:07:17.589083  108432 helper_linux.go:74] Found default gateway interface br-ex 192.168.123.1
F0512 19:07:17.589141  108432 ovnkube.go:133] failed to get default gateway interface
    Copy to Clipboard Toggle word wrap

    The only resolution is to reconfigure the host networking so that both IP families contain the default gateway.

  • If you set the ipv6.disable parameter to 1 in the kernelArgument section of the MachineConfig custom resource (CR) for your cluster, OVN-Kubernetes pods enter a CrashLoopBackOff state. Additionally, updating your cluster to a later version of Red Hat build of MicroShift fails because the Network Operator remains on a Degraded state. Red Hat does not support disabling IPv6 adddresses for your cluster so do not set the ipv6.disable parameter to 1.

Use the ingress controller options in the MicroShift configuration file to make pods and services accessible outside the node.

5.1. Using ingress control in MicroShift
Copy link

When you create your MicroShift node, each pod and service running on the node is allocated an IP address. These IP addresses are accessible to other pods and services running nearby by default, but are not accessible to external clients. MicroShift uses a minimal implementation of the OpenShift Container Platform IngressController API to enable external access to node services.

With more configuration options, you can fine-tune ingress to meet your specific needs. To use enhanced ingress control, update the parameters in the MicroShift configuration file and restart the service. Ingress configuration is useful in a variety of ways, for example:

  • If your application starts processing requests from clients but the connection is closed before it can respond, you can set the ingress.tuningOptions.serverTimeout parameter in the configuration file to a higher value to accommodate the speed of the response from the server.
  • If the router has many connections open because an application running on the node does not close connections properly, you can set the ingress.tuningOptions.serverTimeout and spec.tuningOptions.serverFinTimeout parameters to a lower value, forcing those connections to close sooner.
  • If you need to configure the ingress controller to verify client certificates, you can use the ingress.clientTLS parameter to set a clientCA value, which is a reference to a config map. The config map contains the PEM-encoded CA certificate bundle that is used to verify a client’s certificate. Optionally, you can also configure a list of certificate subject filters.
  • If you need to configure a TLS security profile for an ingress controller, you can use the ingress.tlsSecurityProfile parameter to specify a default or custom individual TLS security profiles. The TLS security profile defines the minimum TLS version and the TLS ciphers for TLS connections for the ingress controllers. If a TLS security profile is not configured, the default value is based on the TLS security profile set for the API server.
  • If you need to define a policy for handling new route claims, you can use the routeAdmission parameter to allow or deny claims across namespaces. You set the routeAdmission parameter to describe how hostname claims across namespaces should be handled and to describe how routes with wildcard policies are handled by the ingress controller.

5.2. Configuring ingress control in MicroShift
Copy link

You can use detailed ingress control settings by updating the MicroShift service configuration file or using a configuration snippet.

Important
  • A config.yaml configuration file takes precedence over built-in settings. The config.yaml file is read every time the MicroShift service starts.
  • Configuration snippet YAMLs take precedence over both built-in settings and the config.yaml configuration file.

Prerequisites

  • You installed the OpenShift CLI (oc).
  • You have root access to the node.
  • Your node uses the OVN-Kubernetes Container Network Interface (CNI) plugin.

Procedure

  1. Apply ingress control settings in one of the two following ways:

    1. Update the MicroShift config.yaml configuration file by making a copy of the provided config.yaml.default file in the /etc/microshift/ directory, naming it config.yaml and keeping it in the source directory.
    2. Use a configuration snippet to apply the ingress control settings you want. To do this, create a configuration snippet YAML file and put it in the /etc/microshift/config.d/ configuration directory.

  2. Replace the default values in the ingress section of the MicroShift YAML with your valid values, or create a configuration snippet file with the sections you need.

    Ingress controller configuration fields with default values

    apiServer:
# ...
ingress:
  certificateSecret: router-certs-custom
  clientTLS:
    allowedSubjectPatterns: []
    clientCA:
      name: ""
    clientCertificatePolicy: ""
  defaultHTTPVersion: 1
  forwardedHeaderPolicy: Append
  httpCompression:
    mimeTypes:
      - ""
  httpEmptyRequestsPolicy: Respond
  listenAddress: []
  logEmptyRequests: Log
  ports:
     http: 80
     https: 443
  routeAdmissionPolicy:
    namespaceOwnership: InterNamespaceAllowed
    wildcardPolicy: WildcardsDisallowed
  status: Managed
  tlsSecurityProfile:
    type:
    custom:
      ciphers:[]
      minTLSVersion:""
    intermediate: {}
    old: {}
  tuningOptions:
    clientFinTimeout: 1s
    clientTimeout: 30s
    headerBufferBytes: 0
    headerBufferMaxRewriteBytes: 0
    healthCheckInterval: 5s
    maxConnections: 0
    serverFinTimeout: 1s
    serverTimeout: 30s
    threadCount: 4
    tlsInspectDelay: 5s
    tunnelTimeout: 1h
# ...
    Copy to Clipboard Toggle word wrap

    Expand
    Table 5.1. Ingress controller configuration fields definitions table
    ParameterDescription

    ingress

    The ingress section of the MicroShift config.yaml file defines the configurable parameters for the implemented portions of the OpenShift Container Platform Ingress Control Operator. All parameters in the rest of this table are subsections in the ingress section of the config.yaml.

    certificateSecret

    A reference to a kubernetes.io/tls type of secret that contains the default certificate that is served by the MicroShift ingress controller. When routes do not specify their own certificate, the certificateSecret parameter is used. All secrets used must contain tls.key key file contents and tls.crt certificate file contents.

    • When the certificateSecret parameter is not set, a wildcard certificate is automatically generated and used. The wildcard certificate is valid for the ingress controller default domain and its subdomains. The generated certificate authority (CA) is automatically integrated with the truststore of the node.
    • In-use generated and user-specified certificates are automatically integrated with the MicroShift built-in OAuth server.

    clientTLS

    Authenticates client access to the node and services. As a result, mutual TLS authentication is enabled. If not set, then client TLS is not enabled. You must set the spec.clientTLS.clientCertificatePolicy and spec.clientTLS.ClientCA parameters to use client TLS.

    clientTLS.AllowedSubjectPatterns

    Optional subfield that specifies a list of regular expressions that are matched against the distinguished name on a valid client certificate to filter requests. Use this parameter to cause the ingress controller to reject certificates based on the distinguished name. The Perl Compatible Regular Expressions (PCRE) syntax is required.

    Important

    When configured, this field must contain a valid expression or the MicroShift service fails. At least one pattern must match a client certificate’s distinguished name; otherwise, the ingress controller rejects the certificate and denies the connection.

    clientTLS.clientCA

    Specifies a required config map that is in the openshift-ingress namespace. Required to enable client TLS. The config map must contain a certificate authority (CA) bundle named ca-bundle.pem or the deployment of the default router fails.

    clientTLS.ClientCA.name

    The metadata.name of the config map referenced in the clientTLS.ClientCA value.

    clientTLS.ClientCertificatePolicy

    Required or Optional are valid values. Set to Required to enable client TLS. The ingress controller only checks client certificates for edge-terminated and re-encrypted TLS routes. The ingress controller cannot check certificates for plain text HTTP or passthrough TLS routes.

    defaultHTTPVersion

    Sets the HTTP version for the ingress controller. Default value is 1 for HTTP 1.1.

    forwardedHeaderPolicy

    Specifies when and how the ingress controller sets the Forwarded, X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Port, X-Forwarded-Proto, and X-Forwarded-Proto-Version HTTP headers. The following values are valid:

    • Append preserves any existing headers by specifying that the ingress controller appends them. 'Append` is the default value.
    • Replace removes any existing headers by specifying that the ingress controller sets the headers.
    • IfNone sets the headers set by specifying that the ingress controller sets the headers if they are not already set.
    • Never preserves any existing headers by specifying that the ingress controller never sets the headers.

    httpCompression

    Defines the policy for HTTP traffic compression.

    httpCompression.mimeTypes

    Defines a list of MIME types to which compression should be applied.

    • For example, text/css; charset=utf-8, text/html, text/*, image/svg+xml, application/octet-stream, X-custom/customsub, in the, type/subtype; [;attribute=value] format.
    • Valid types are: application, image, message, multipart, text, video, or a custom type prefaced by X-. To see the full notation for MIME types and subtypes, see RFC1341 (IETF Datatracker documentation).

    httpEmptyRequestsPolicy

    Describes how HTTP connections are handled if the connection times out before a request is received. Allowed values for this field are Respond and Ignore. The default value is Respond. Empty requests typically come from load-balancer health probes or preconnects and can often be safely ignored. However, these requests can also be caused by network errors and port scans. Therefore, setting this field to Ignore can impede detection or diagnosis of network problems and detecting intrusion attempts.

    • When the policy is set to Respond, the ingress controller sends an HTTP 400 or 408 response, logs the connection if access logging is enabled, and counts the connection in the appropriate metrics.
    • When the policy is set to Ignore, the http-ignore-probes parameter is added to the HAproxy process configuration. After this parameter is added, the ingress controller closes the connection without sending a response, then either logs the connection or incrementing metrics.

    logEmptyRequests

    Specifies connections for which no request is received and logged. Log and Ignore are valid values. Empty requests typically come from load-balancer health probes or preconnects and can often be safely ignored. However, these requests can also be caused by network errors and port scans. Therefore, setting this field to Ignore can impede detection or diagnosis of network problems and detecting intrusion attempts. The default value is Log.

    • Setting this value to Log indicates that an event should be logged.
    • Setting this value to Ignore sets the dontlognull option in the HAproxy configuration.

    ports

    Defines default router ports.

    ports.http

    Default router http port. Must be in range 1-65535. Default value is 80.

    ports.https

    Default router https port. Must be in range 1-65535. Default value is 443.

    routeAdmission

    Defines a policy for handling new route claims, such as allowing or denying claims across namespaces.

    routeAdmission.namespaceOwnership

    Describes how hostname claims across namespaces should be handled. The default is InterNamespaceAllowed. The following are valid values:

    • Strict does not allow routes to claim the same hostname across namespaces.
    • InterNamespaceAllowed allows routes to claim different paths of the same hostname across namespaces.

    routeAdmission.wildcardPolicy

    Controls how routes with configured wildcard policies are handled by the ingress controller. WildcardsAllowed and WildcardsDisallowed are valid values. Default value is WildcardsDisallowed.

    • WildcardPolicyAllowed means that routes with any wildcard policy are admitted by the ingress controller.
    • WildcardPolicyDisallowed means that only routes with a wildcard policy of None are admitted by the ingress controller.
    Important

    Changing the wildcard policy from WildcardsAllowed to WildcardsDisallowed causes admitted routes with a wildcard policy of subdomain to stop working. These routes must be recreated to a wildcard policy of None to be readmitted by the ingress controller.

    status

    Default router status. Managed or Removed are valid values.

    tlsSecurityProfile

    tlsSecurityProfile specifies settings for TLS connections for ingress controllers. If not set, the default value is based on the apiservers.config.openshift.io/cluster resource. The TLS 1.0 version of an Old or Custom profile is automatically converted to 1.1 by the ingress controller. Intermediate is the default setting.

    • The minimum TLS version for ingress controllers is 1.1. The maximum TLS version is 1.3.
    Note

    Ciphers and the minimum TLS version of the configured security profile are reflected in the TLSProfile status. Profiles are intent-based and change over time when new ciphers are developed and existing ciphers are found to be insecure. The usable list can be reduced depending on which ciphers are available to a specific process.

    tlsSecurityProfile.custom

    User-defined TLS security profile. If you configure this parameter and related parameters, use extreme caution.

    tlsSecurityProfile.custom.ciphers

    Specifies the cipher algorithms that are negotiated during the TLS handshake. Operators might remove entries their operands do not support.

    tlsSecurityProfile.custom.minTLSVersion

    Specifies the minimal version of the TLS protocol that is negotiated during the TLS handshake. For example, to use TLS versions 1.1, 1.2 and 1.3, set the value to VersionTLS11. The highest valid value for minTLSVersion is VersionTLS12.

    tlsSecurityProfile.intermediate

    This TLS profile can be used for a majority of services. Intermediate compatibility (recommended).

    tlsSecurityProfile.old

    Used for backward compatibility. Old backward compatibility.

    tlsSecurityProfile.type

    Valid values are Intermediate, Old, or Custom. The Modern value is not supported.

    tuningOptions

    Specifies options for tuning the performance of ingress controller pods.

    tuningOptions.clientFinTimeout

    Specifies how long a connection is held open while waiting for the client response to the server closing the connection. The default timeout is 1s.

    tuningOptions.clientTimeout

    Specifies how long a connection is held open while waiting for a client response. The default timeout is 30s.

    tuningOptions.headerBufferBytes

    Specifies how much memory is reserved, in bytes, for ingress controller connection sessions. This value must be at least 16384 if HTTP/2 is enabled for the ingress controller. If not set, the default value is 32768 bytes.

    Important

    Setting this field not recommended because headerBufferMaxRewriteBytes parameter values that are too small can break the ingress controller. Conversely, values for headerBufferMaxRewriteBytes that are too large could cause the ingress controller to use significantly more memory than necessary.

    tuningOptions.headerBufferMaxRewriteBytes

    Specifies how much memory should be reserved, in bytes, from headerBufferBytes for HTTP header rewriting and appending for ingress controller connection sessions. The minimum value for headerBufferMaxRewriteBytes is 4096. headerBufferBytes must be greater than the headerBufferMaxRewriteBytes value for incoming HTTP requests. If not set, the default value is 8192 bytes.

    Important

    Setting this field is not recommended because headerBufferMaxRewriteBytes values that are too small can break the ingress controller and headerBufferMaxRewriteBytes that are too large could cause the ingress controller to use significantly more memory than necessary.

    tuningOptions.healthCheckInterval

    Specifies how long the router waits between health checks, set in seconds. The default is 5s.

    tuningOptions.maxConnections

    Specifies the maximum number of simultaneous connections that can be established for each HAProxy process. Increasing this value allows each ingress controller pod to handle more connections at the cost of additional system resources. Permitted values are 0, -1, any value within the range 2000 and 2000000, or the field can be left empty.

    • If this field is left empty or has the value 0, the ingress controller uses the default value of 50000.
    • If the field has the value of -1, then the HAProxy process dynamically computes a maximum value based on the available ulimits in the running container. This process results in a large computed value that incurs significant memory usage compared to the current default value of 50000.
    • If the field has a value that is greater than the current operating system limit, the HAProxy processes do not start.
    • If you choose a discrete value and the router pod is migrated to a new node, it is possible that the new node does not have an identical ulimit configured. In such cases, the pod fails to start.
    • If you have nodes with different ulimits configured, and you choose a discrete value, you can use the value of -1 for this field so that the maximum number of connections is calculated at runtime.
    • You can monitor memory usage for router containers with the container_memory_working_set_bytes{container="router",namespace="openshift-ingress"} metric.
    • You can monitor memory usage of individual HAProxy processes in router containers with the container_memory_working_set_bytes{container="router",namespace="openshift-ingress"}/container_processes{container="router",namespace="openshift-ingress"} metric.

    tuningOptions.serverFinTimeout

    Specifies how long a connection is held open while waiting for the server response to the client that is closing the connection. The default timeout is 1s.

    tuningOptions.serverTimeout

    Specifies how long a connection is held open while waiting for a server response. The default timeout is 30s.

    tuningOptions.threadCount

    Specifies the number of threads to create per HAProxy process. Creating more threads allows each ingress controller pod to handle more connections, at the cost of more system resources being used. The HAProxy load balancer supports up to 64 threads. If this field is empty, the ingress controller uses the default value of 4 threads.

    Important

    Setting this field is not recommended because increasing the number of HAProxy threads allows ingress controller pods to use more CPU time under load, and prevent other pods from receiving the CPU resources they need to perform. Reducing the number of threads can cause the ingress controller to perform poorly.

    tuningOptions.tlsInspectDelay

    Specifies how long the router can hold data to find a matching route. Setting this value too low can cause the router to fall back to the default certificate for edge-terminated, re-encrypted, or passthrough routes, even when using a better-matched certificate. The default inspect delay is 5s.

    tuningOptions.tunnelTimeout

    Specifies how long a tunnel connection, including websockets, remains open while the tunnel is idle. The default timeout is 1h.

  3. Complete any other configurations you require, then start or restart MicroShift by running one the following commands: 

    $ sudo systemctl start microshift
    Copy to Clipboard Toggle word wrap 
    $ sudo systemctl restart microshift
    Copy to Clipboard Toggle word wrap

Verification

After making ingress configuration changes and restarting MicroShift, you can check the age of the router pod to ensure that changes have been applied.

  • To check the status of the router pod, run the following command: 

    $ oc get pods -n openshift-ingress
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                              READY   STATUS    RESTARTS   AGE
router-default-8649b5bf65-w29cn   1/1     Running   0          6m10s
    Copy to Clipboard Toggle word wrap

Use this procedure to create a secret that is referenced by the certificateSecret parameter value in the MicroShift configuration file. This secret contains the default certificate served by the ingress controller.

Note

Any in-use certificates is automatically integrated with the MicroShift built-in OAuth server.

Prerequisites

  • You have root access to MicroShift.
  • You installed the OpenShift CLI (oc).
  • Your private key is not encrypted or you have decrypted it for importing into MicroShift.

Procedure

  1. Create a secret that contains the wildcard certificate chain and key: 

    $ oc create secret tls <secret>
    1 
    
     --cert=</path/to/cert.crt>
    2 
    
     --key=</path/to/cert.key>
    3 
    
     -n openshift-ingress
    Copy to Clipboard Toggle word wrap
    1
    <secret> is the name of the secret that contains the certificate chain and private key.
    2
    </path/to/cert.crt> is the path to the certificate chain on your local file system.
    3
    </path/to/cert.key> is the path to the private key associated with this certificate.
    Important

    The certificate must include the subjectAltName extension showing *.apps.<nodename>.<domain>.

  2. Update the certificateSecret parameter value in the MicroShift configuration YAML with the newly created secret.

  3. Complete any other configurations you require, then start or restart MicroShift by running one the following commands: 

    $ sudo systemctl start microshift
    Copy to Clipboard Toggle word wrap 
    $ sudo systemctl restart microshift
    Copy to Clipboard Toggle word wrap

You can configure the TLS security profile for the ingress controller to use by setting the type in MicroShift configuration YAML.

Prerequisites

  • You have root access to the MicroShift node.

Procedure

  1. Add the spec.tlsSecurityProfile field to the MicroShift YAML configuration file. 

     ...
spec:
  tlsSecurityProfile:
    type: Custom
    1 
    
    custom:
    2 
    
      ciphers:
    3 
    
      - ECDHE-ECDSA-CHACHA20-POLY1305
      - ECDHE-RSA-CHACHA20-POLY1305
      - ECDHE-RSA-AES128-GCM-SHA256
      - ECDHE-ECDSA-AES128-GCM-SHA256
      minTLSVersion: VersionTLS11
 ...
    Copy to Clipboard Toggle word wrap
    1
    Specify the TLS security profile type (Old, Intermediate, or Custom). The default is Intermediate.
    2
    Specify the appropriate field for the selected type:
    • old: {}
    • intermediate: {}
    • custom:
    3
    For the custom type, specify a list of TLS ciphers and minimum accepted TLS version.
    Warning

    If you choose a custom TLS configuration, use extreme caution. Using self-signed TLS certificates can introduce security risks.

  2. Save the file to apply the changes.

  3. Restart MicroShift by running the following command: 

    $ sudo systemctl restart microshift
    Copy to Clipboard Toggle word wrap

You can configure MicroShift to disable the built-in logical volume manager storage (LVMS) Container Storage Interface (CSI) provider or the CSI snapshot capabilities to reduce the use of runtime resources such as RAM, CPU, and storage.

Use the following procedure to disable installation of the CSI implementation pods.

Important

This procedure is for users who are defining the configuration file before installing and running MicroShift. If MicroShift is already started then CSI snapshot implementation will be running. Users must manually remove it by following the uninstallation instructions.

Note

MicroShift will not delete CSI snapshot implementation pods. You must configure MicroShift to disable installation of the CSI snapshot implementation pods during the startup process.

Procedure

  1. Disable installation of the CSI snapshot controller by entering the optionalCsiComponents value under the storage section of the MicroShift configuration file in /etc/microshift/config.yaml: 

    # ...
  storage: {}
    1 
    
# ...
    Copy to Clipboard Toggle word wrap
    1
    Accepted values are:
    • Not defining optionalCsiComponents.
    • Specifying optionalCsiComponents field with an empty value ([]) or a single empty string element ([""]).

    • Specifying optionalCsiComponents with one of the accepted values which are snapshot-controller, or none. A value of none is mutually exclusive with all other values.

      Note

      If the optionalCsiComponents value is empty or null, MicroShift defaults to deploying snapshot-controller.

  2. After the optionalCsiComponents field is specified with a supported value in the config.yaml, start MicroShift by running the following command: 

    $ sudo systemctl start microshift
    Copy to Clipboard Toggle word wrap
    Note

    MicroShift does not redeploy the disabled components after a restart.

Use the following procedure to disable installation of the CSI implementation pods. MicroShift does not delete CSI driver implementation pods. You must configure MicroShift to disable installation of the CSI driver implementation pods during the startup process.

Important

This procedure is for defining the configuration file before installing and running MicroShift. If MicroShift is already started, then the CSI driver implementation is running. You must manually remove it by following the uninstallation instructions.

Procedure

  1. Disable installation of the CSI driver by entering the driver value under the storage section of the MicroShift configuration file in /etc/microshift/config.yaml: 

    # ...
  storage
   driver:
   - "none"
    1 
    
# ...
    Copy to Clipboard Toggle word wrap
    1
    Valid values are none or lvms.
    Note

    By default, the driver value is empty or null and LVMS is deployed.

  2. Start MicroShift after the driver field is specified with a supported value in the /etc/microshift/config.yaml file by running the following command: 

    $ sudo systemctl enable --now microshift
    Copy to Clipboard Toggle word wrap
    Note

    MicroShift does not redeploy the disabled components after a restart operation.

Chapter 7. Checking greenboot scripts status
Copy link

To deploy applications or make other changes through the MicroShift API with tools other than kustomize manifests, you must wait until the greenboot health checks have finished. This ensures that your changes are not lost if greenboot rolls your rpm-ostree system back to an earlier state.

The greenboot-healthcheck service runs one time and then exits. After greenboot has exited and the system is in a healthy state, you can proceed with configuration changes and deployments.

Check the status of greenboot health checks before making changes to the system or during troubleshooting. You can use any of the following commands to help you ensure that greenboot scripts have finished running.

Procedure

  • To see a report of health check status, use the following command: 

    $ systemctl show --property=SubState --value greenboot-healthcheck.service
    Copy to Clipboard Toggle word wrap
    • An output of start means that greenboot checks are still running.
    • An output of exited means that checks have passed and greenboot has exited. Greenboot runs the scripts in the green.d directory when the system is a healthy state.
    • An output of failed means that checks have not passed. Greenboot runs the scripts in red.d directory when the system is in this state and might restart the system.

  • To see a report showing the numerical exit code of the service where 0 means success and non-zero values mean a failure occurred, use the following command: 

    $ systemctl show --property=ExecMainStatus --value greenboot-healthcheck.service
    Copy to Clipboard Toggle word wrap

  • To see a report showing a message about boot status, such as Boot Status is GREEN - Health Check SUCCESS, use the following command: 

    $ cat /run/motd.d/boot-status
    Copy to Clipboard Toggle word wrap

Chapter 8. Node access with kubeconfig files
Copy link

Learn about how kubeconfig files are used with MicroShift deployments. CLI tools use kubeconfig files to communicate with the API server of a node. These files provide node details, IP addresses, and other information needed for authentication.

8.1. Kubeconfig files for configuring node access
Copy link

The two categories of kubeconfig files used in MicroShift are local access and remote access. Every time MicroShift starts, a set of kubeconfig files for local and remote access to the API server are generated. These files are generated in the /var/lib/microshift/resources/kubeadmin/ directory by using preexisting configuration information.

Each access type requires a different authentication certificate signed by different Certificate Authorities (CAs). The generation of multiple kubeconfig files accommodates this need.

You can use the appropriate kubeconfig file for the access type needed in each case to provide authentication details. The contents of MicroShift kubeconfig files are determined by either default built-in values or a config.yaml file.

Note

A kubeconfig file must exist for the cluster to be accessible. The values are applied from built-in default values or a customized config.yaml file.

Example contents of the kubeconfig files

/var/lib/microshift/resources/kubeadmin/
├── kubeconfig
1 

├── alt-name-1
2 

│   └── kubeconfig
├── 1.2.3.4
3 

│   └── kubeconfig
└── microshift-rhel9
4 

    └── kubeconfig
Copy to Clipboard Toggle word wrap

1
Local hostname. The main IP address of the host is always the default.
2
Subject Alternative Names for API server certificates.
3
DNS name.
4
MicroShift hostname.

8.2. Local access kubeconfig file
Copy link

The local access kubeconfig file is written to /var/lib/microshift/resources/kubeadmin/kubeconfig. This kubeconfig file provides access to the API server by using localhost. Choose this file when you are connecting the node locally.

Example contents of kubeconfig for local access

clusters:
- cluster:
    certificate-authority-data: <base64 CA>
    server: https://localhost:6443
Copy to Clipboard Toggle word wrap

The localhost kubeconfig file can only be used from a client connecting to the API server from the same host. The certificates in the file do not work for remote connections.

8.2.1. Accessing the MicroShift node locally
Copy link

Use the following procedure to access the MicroShift node locally by using a kubeconfig file.

Prerequisites

  • You installed OpenShift CLI (oc).

Procedure

  1. Optional: to create a ~/.kube/ folder if your Red Hat Enterprise Linux (RHEL) machine does not have one, run the following command: 

    $ mkdir -p ~/.kube/
    Copy to Clipboard Toggle word wrap

  2. Copy the generated local access kubeconfig file to the ~/.kube/ directory by running the following command: 

    $ sudo cat /var/lib/microshift/resources/kubeadmin/kubeconfig > ~/.kube/config
    Copy to Clipboard Toggle word wrap

  3. Update the permissions on your ~/.kube/config file by running the following command: 

    $ chmod go-r ~/.kube/config
    Copy to Clipboard Toggle word wrap

Verification

  • Verify that MicroShift is running by entering the following command: 

    $ oc get pods -A
    Copy to Clipboard Toggle word wrap

    Example output

    NAMESPACE                   NAME                                                     READY   STATUS   RESTARTS  AGE
default                     i-06166fbb376f14a8bus-west-2computeinternal-debug-qtwcr  1/1     Running  0		    46m
kube-system                 csi-snapshot-controller-5c6586d546-lprv4                 1/1     Running  0		    51m
openshift-dns               dns-default-45jl7                                        2/2     Running  0		    50m
openshift-dns               node-resolver-7wmzf                                      1/1     Running  0		    51m
openshift-ingress           router-default-78b86fbf9d-qvj9s                          1/1     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-master-5rfhh                                     4/4     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-node-gcnt6                                       1/1     Running  0		    51m
openshift-service-ca        service-ca-bf5b7c9f8-pn6rk                               1/1     Running  0		    51m
openshift-storage           topolvm-controller-549f7fbdd5-7vrmv                      5/5     Running  0		    51m
openshift-storage           topolvm-node-rht2m                                       3/3     Running  0		    50m
    Copy to Clipboard Toggle word wrap

    Note

    This example output shows basic MicroShift. If you have installed optional RPMs, the status of pods running those services is also expected to be shown in your output.

8.3. Remote access kubeconfig files
Copy link

When a MicroShift node connects to the API server from an external source, a certificate with all of the alternative names in the SAN field is used for validation. MicroShift generates a default kubeconfig for external access by using the hostname value. The defaults are set in the <node.hostnameOverride>, <node.nodeIP> and api.<dns.baseDomain> parameter values of the default kubeconfig file.

The /var/lib/microshift/resources/kubeadmin/<hostname>/kubeconfig file uses the hostname of the machine, or node.hostnameOverride if that option is set, to reach the API server. The CA of the kubeconfig file is able to validate certificates when accessed externally.

Example contents of a default kubeconfig file for remote access

clusters:
- cluster:
    certificate-authority-data: <base64 CA>
    server: https://microshift-rhel9:6443
Copy to Clipboard Toggle word wrap

8.3.1. Remote access customization
Copy link

Multiple remote access kubeconfig file values can be generated for accessing the node with different IP addresses or host names. An additional kubeconfig file generates for each entry in the apiServer.subjectAltNames parameter. You can copy remote access kubeconfig files from the host during times of IP connectivity and then use them to access the API server from other workstations.

You can generate additional kubeconfig files to use if you need more host names or IP addresses than the default remote access file provides.

Important

You must restart MicroShift for configuration changes to be implemented.

Prerequisites

  • You have created a config.yaml for MicroShift.

Procedure

  1. Optional: You can show the contents of the config.yaml. Run the following command: 

    $ cat /etc/microshift/config.yaml
    Copy to Clipboard Toggle word wrap

  2. Optional: You can show the contents of the remote-access kubeconfig file. Run the following command: 

    $ cat /var/lib/microshift/resources/kubeadmin/<hostname>/kubeconfig
    Copy to Clipboard Toggle word wrap
    Important

    Additional remote access kubeconfig files must include one of the server names listed in the Red Hat build of MicroShift config.yaml file. Additional kubeconfig files must also use the same CA for validation.

  3. To generate additional kubeconfig files for additional DNS names SANs or external IP addresses, add the entries you need to the apiServer.subjectAltNames field. In the following example, the DNS name used is alt-name-1 and the IP address is 1.2.3.4.

    Example config.yaml with additional authentication values

    dns:
  baseDomain: example.com
node:
  hostnameOverride: "microshift-rhel9"
    1 
    
  nodeIP: 10.0.0.1
apiServer:
  subjectAltNames:
  - alt-name-1
    2 
    
  - 1.2.3.4
    3
    Copy to Clipboard Toggle word wrap

    1
    Hostname
    2
    DNS name
    3
    IP address or range

  4. Restart MicroShift to apply configuration changes and auto-generate the kubeconfig files you need by running the following command: 

    $ sudo systemctl restart microshift
    Copy to Clipboard Toggle word wrap

  5. To check the contents of additional remote-access kubeconfig files, insert the name or IP address as listed in the config.yaml into the cat command. For example, alt-name-1 is used in the following example command: 

    $ cat /var/lib/microshift/resources/kubeadmin/alt-name-1/kubeconfig
    Copy to Clipboard Toggle word wrap

  6. Choose the kubeconfig file to use that contains the SAN or IP address you want to use to connect your node. In this example, the kubeconfig containing`alt-name-1` in the cluster.server field is the correct file.

    Example contents of an additional kubeconfig file

    clusters:
- cluster:
    certificate-authority-data: <base64 CA>
    server: https://alt-name-1:6443
    1
    Copy to Clipboard Toggle word wrap

    1
    The /var/lib/microshift/resources/kubeadmin/alt-name-1/kubeconfig file values are from the apiServer.subjectAltNames configuration values.
Note

All of these parameters are included as common names (CN) and subject alternative names (SAN) in the external serving certificates for the API server.

Use the following procedure to open the firewall so that a remote user can access the MicroShift service. You must complete this procedure before a workstation user can access the node remotely.

For this procedure, user@microshift is the user on the MicroShift host machine and is responsible for setting up that machine so that it can be accessed by a remote user on a separate workstation.

Prerequisites

  • You installed OpenShift CLI (oc).
  • Your account has cluster administration privileges.

Procedure

  • As user@microshift on the MicroShift host, open the firewall port for the Kubernetes API server (6443/tcp) by running the following command: 

    [user@microshift]$ sudo firewall-cmd --permanent --zone=public --add-port=6443/tcp && sudo firewall-cmd --reload
    Copy to Clipboard Toggle word wrap

Verification

  • As user@microshift, verify that MicroShift is running by entering the following command: 

    $ oc get pods -A
    Copy to Clipboard Toggle word wrap

    Example output

    NAMESPACE                   NAME                                                     READY   STATUS   RESTARTS  AGE
default                     i-06166fbb376f14a8bus-west-2computeinternal-debug-qtwcr  1/1     Running  0		    46m
kube-system                 csi-snapshot-controller-5c6586d546-lprv4                 1/1     Running  0		    51m
openshift-dns               dns-default-45jl7                                        2/2     Running  0		    50m
openshift-dns               node-resolver-7wmzf                                      1/1     Running  0		    51m
openshift-ingress           router-default-78b86fbf9d-qvj9s                          1/1     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-master-5rfhh                                     4/4     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-node-gcnt6                                       1/1     Running  0		    51m
openshift-service-ca        service-ca-bf5b7c9f8-pn6rk                               1/1     Running  0		    51m
openshift-storage           topolvm-controller-549f7fbdd5-7vrmv                      5/5     Running  0		    51m
openshift-storage           topolvm-node-rht2m                                       3/3     Running  0		    50m
    Copy to Clipboard Toggle word wrap

    Note

    This example output shows basic MicroShift. If you have installed optional RPMs, the status of pods running those services is also expected to be shown in your output.

8.4.2. Accessing the MicroShift node remotely
Copy link

Use the following procedure to access the MicroShift service from a remote location by using a kubeconfig file.

The user@workstation login is used to access the host machine remotely. The <user> value in the procedure is the name of the user that user@workstation logs in with to the MicroShift host.

Prerequisites

  • You installed OpenShift CLI (oc).
  • The user@microshift has opened the firewall from the local host.
  • You generated additional kubeconfig files.

Procedure

  1. As user@workstation, create a ~/.kube/ folder if your Red Hat Enterprise Linux (RHEL) machine does not have one by running the following command: 

    [user@workstation]$ mkdir -p ~/.kube/
    Copy to Clipboard Toggle word wrap

  2. As user@workstation, set a variable for the hostname of your MicroShift host by running the following command: 

    [user@workstation]$ MICROSHIFT_MACHINE=<microshift_hostname>
    1
    Copy to Clipboard Toggle word wrap
    1
    Replace the value, <MicroShift_hostname>, with the either the name or the IP address of the host running .

  3. As user@workstation, copy the generated kubeconfig file that contains the hostname or IP address you want to connect to from the RHEL machine running MicroShift to your local machine by running the following command: 

    [user@workstation]$ ssh <user>@$MICROSHIFT_MACHINE "sudo cat /var/lib/microshift/resources/kubeadmin/$MICROSHIFT_MACHINE/kubeconfig" > ~/.kube/config
    1
    Copy to Clipboard Toggle word wrap
    1
    Replace <user> with your SSH login credentials.

  4. As user@workstation, update the permissions on your ~/.kube/config file by running the following command: 

    $ chmod go-r ~/.kube/config
    Copy to Clipboard Toggle word wrap

Verification

  • As user@workstation, verify that MicroShift is running by entering the following command: 

    $ oc get pods -A
    Copy to Clipboard Toggle word wrap

    Example output

    NAMESPACE                   NAME                                                     READY   STATUS   RESTARTS  AGE
default                     i-06166fbb376f14a8bus-west-2computeinternal-debug-qtwcr  1/1     Running  0		    46m
kube-system                 csi-snapshot-controller-5c6586d546-lprv4                 1/1     Running  0		    51m
openshift-dns               dns-default-45jl7                                        2/2     Running  0		    50m
openshift-dns               node-resolver-7wmzf                                      1/1     Running  0		    51m
openshift-ingress           router-default-78b86fbf9d-qvj9s                          1/1     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-master-5rfhh                                     4/4     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-node-gcnt6                                       1/1     Running  0		    51m
openshift-service-ca        service-ca-bf5b7c9f8-pn6rk                               1/1     Running  0		    51m
openshift-storage           topolvm-controller-549f7fbdd5-7vrmv                      5/5     Running  0		    51m
openshift-storage           topolvm-node-rht2m                                       3/3     Running  0		    50m
    Copy to Clipboard Toggle word wrap

    Note

    This example output shows basic MicroShift. If you have installed optional RPMs, the status of pods running those services is also expected to be shown in your output.

9.1. Configuring custom certificate authorities
Copy link

Allow and encrypt connections with external clients by replacing the MicroShift default API server certificate with a custom server certificate issued by a certificate authority (CA).

When MicroShift starts, an internal MicroShift node certificate authority (CA) issues the default API server certificate. By default, clients outside of the node cannot verify the MicroShift-issued API server certificate. You can grant secure access and encrypt connections between the MicroShift API server and external clients. Replace the default certificate with a custom server certificate issued externally by a CA that clients trust.

The following steps illustrate the workflow for customizing the API server certificate configuration in MicroShift:

  1. Copy the certificates and keys to the preferred directory in the host operating system. Ensure that the files are accessible only with root access.

  2. Update the MicroShift configuration for each custom CA by specifying the certificate names and new fully qualified domain name (FQDN) in the MicroShift /etc/microshift/config.yaml configuration file.

    Each certificate configuration can contain the following values:

    • The certificate file location is a required value.

    • A single common name containing the API server DNS and IP address or IP address range.

      Tip

      In most cases, MicroShift generates a new kubeconfig file for your custom CA that includes the IP address or range that you specify. The exception is when you specify wildcards for the IP address. In this case, MicroShift generates a kubeconfig file with the public IP address of the server. To use wildcards, you must update the kubeconfig file with your specific details.

    • Multiple Subject Alternative Names (SANs) containing the API server DNS and IP addresses or a wildcard certificate.
    • You can list additional DNS names for each certificate.
  3. After the MicroShift service restarts, you must copy the generated kubeconfig files to the client.

  4. Configure additional CAs on the client system. For example, you can update CA bundles in the Red Hat Enterprise Linux (RHEL) truststore.

    Important

    Custom server certificates must be validated against CA data configured in the trust root of the host operating system. For more information, read the following documentation:

  5. The certificates and keys are read from the specified file location on the host. You can test and validate configuration from the client.

    • If any validation fails, MicroShift skips the custom configuration and uses the default certificate to start. The priority is to continue the service uninterrupted. MicroShift logs errors when the service starts. Common errors include expired certificates, missing files, or wrong IP addresses.
  6. External server certificates are not automatically renewed. You must manually rotate your external certificates.

9.1.2. Configuring custom certificate authorities
Copy link

To configure externally generated certificates and domain names by using custom certificate authorities (CAs), add them to the MicroShift /etc/microshift/config.yaml configuration file. You must also configure the host operating system trust root.

Note

Externally generated kubeconfig files are created in the /var/lib/microshift/resources/kubeadmin/<hostname>/kubeconfig directory. If you need to use localhost in addition to externally generated configurations, retain the original kubeconfig file in its default location. The localhost kubeconfig file uses the self-signed certificate authority.

Prerequisites

  • The OpenShift CLI (oc) is installed.
  • You have root access to the node.
  • The certificate authority has issued the custom certificates.
  • A MicroShift /etc/microshift/config.yaml configuration file exists.

Procedure

  1. Copy the custom certificates you want to add to the trust root of the MicroShift host. Ensure that the certificate and private keys are only accessible to MicroShift.

  2. For each custom CA that you need, add an apiServer section called namedCertificates to the /etc/microshift/config.yaml MicroShift configuration file by using the following example: 

    apiServer:
  namedCertificates:
   - certPath: ~/certs/api_fqdn_1.crt
    1 
    
     keyPath:  ~/certs/api_fqdn_1.key
    2 
    
   - certPath: ~/certs/api_fqdn_2.crt
     keyPath:  ~/certs/api_fqdn_2.key
     names:
    3 
    
     - api_fqdn_1
     - *.apps.external.com
    Copy to Clipboard Toggle word wrap
    1
    Add the full path to the certificate.
    2
    Add the full path to the certificate key.
    3
    Optional. Add a list of explicit DNS names. Leading wildcards are allowed. If no names are listed, the implicit names are extracted from the certificates.

  3. Restart the MicroShift to apply the certificates by running the following command: 

    $ systemctl microshift restart
    Copy to Clipboard Toggle word wrap
  4. Wait a few minutes for the system to restart and apply the custom server. New kubeconfig files are generated in the /var/lib/microshift/resources/kubeadmin/ directory.
  5. Copy the kubeconfig files to the client. If you specified wildcards for the IP address, update the kubeconfig to remove the public IP address of the server and replace that IP address with the specific wildcard range you want to use.

  6. From the client, use the following steps:

    1. Specify the kubeconfig to use by running the following command: 

      $ export KUBECONFIG=~/custom-kubeconfigs/kubeconfig
      1
      Copy to Clipboard Toggle word wrap
      1
      Use the location of the copied kubeconfig file as the path.

    2. Check that the certificates are applied by using the following command: 

      $ oc --certificate-authority ~/certs/ca.ca get node
      Copy to Clipboard Toggle word wrap

      Example output

      oc get node
NAME                             STATUS   ROLES                         AGE   VERSION
dhcp-1-235-195.arm.example.com   Ready    control-plane,master,worker   76m   v1.32.3
      Copy to Clipboard Toggle word wrap

    3. Add the new CA file to the $KUBECONFIG environment variable by running the following command: 

      $ oc config set clusters.microshift.certificate-authority /tmp/certificate-authority-data-new.crt
      Copy to Clipboard Toggle word wrap

    4. Verify that the new kubeconfig file contains the new CA by running the following command: 

      $ oc config view --flatten
      Copy to Clipboard Toggle word wrap

      Example externally generated kubeconfig file

      apiVersion: v1
clusters:
- cluster:
    certificate-authority: /tmp/certificate-authority-data-new.crt
      1 
      
    server: https://api.ci-ln-k0gim2b-76ef8.aws-2.ci.openshift.org:6443
  name: ci-ln-k0gim2b-76ef8
contexts:
- context:
    cluster: ci-ln-k0gim2b-76ef8
    user:
  name:
current-context:
kind: Config
preferences: {}
      Copy to Clipboard Toggle word wrap

      1
      The certificate-authority-data section is not present in externally generated kubeconfig files. It is added with the oc config set command used previously.

    5. Verify the subject and issuer of your customized API server certificate authority by running the following command: 

      $ curl --cacert /tmp/caCert.pem https://${fqdn_name}:6443/healthz -v
      Copy to Clipboard Toggle word wrap

      Example output

      Server certificate:
  subject: CN=kas-test-cert_server
  start date: Mar 12 11:39:46 2024 GMT
  expire date: Mar 12 11:39:46 2025 GMT
  subjectAltName: host "dhcp-1-235-3.arm.eng.rdu2.redhat.com" matched cert's "dhcp-1-235-3.arm.eng.rdu2.redhat.com"
  issuer: CN=kas-test-cert_ca
  SSL certificate verify ok.
      Copy to Clipboard Toggle word wrap

      Important

      Either replace the certificate-authority-data in the generated kubeconfig file with the new rootCA or add the certificate-authority-data to the trust root of the operating system. Do not use both methods.

    6. Configure additional CAs in the trust root of the operating system. For example, in the RHEL Client truststore on the client system. The system-wide truststore.

      • Updating the certificate bundle with the configuration that contains the CA is recommended.
      • If you do not want to configure your certificate bundles, you can alternately use the oc login localhost:8443 --certificate-authority=/path/to/cert.crt command, but this method is not preferred.

9.1.3. Custom certificates reserved name values
Copy link

The following certificate problems cause MicroShift to ignore certificates dynamically and log an error:

  • The certificate files do not exist on the disk or are not readable.
  • The certificate is not parsable.
  • The certificate overrides the internal certificates IP addresses or DNS names in a SubjectAlternativeNames (SAN) field. Do not use a reserved name when configuring SANs.
Expand
Table 9.1. Reserved Names values
AddressTypeComment

localhost

DNS

 

127.0.0.1

IP Address

 

10.42.0.0

IP Address

Node Network

10.43.0.0/16,10.44.0.0/16

IP Address

Service Network

169.254.169.2/29

IP Address

br-ex Network

kubernetes.default.svc

DNS

 

openshift.default.svc

DNS

 

svc.cluster.local

DNS

 

9.1.4. Troubleshooting custom certificates
Copy link

To troubleshoot the implementation of custom certificates, you can take the following steps.

Procedure

  1. From MicroShift, ensure that the certificate is served by the kube-apiserver and verify that the certificate path is appended to the --tls-sni-cert-key FLAG by running the following command: 

    $ journalctl -u microshift -b0 | grep tls-sni-cert-key
    Copy to Clipboard Toggle word wrap

    Example output

    Jan 24 14:53:00 localhost.localdomain microshift[45313]: kube-apiserver I0124 14:53:00.649099   45313 flags.go:64] FLAG: --tls-sni-cert-key="[/home/eslutsky/dev/certs/server.crt,/home/eslutsky/dev/certs/server.key;/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.key;/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.key;/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.key
    Copy to Clipboard Toggle word wrap

  2. From the client, ensure that the kube-apiserver is serving the correct certificate by running the following command: 

    $ openssl s_client -connect <SNI_ADDRESS>:6443 -showcerts | openssl x509 -text -noout -in - | grep -C 1 "Alternative\|CN"
    Copy to Clipboard Toggle word wrap

To stop the MicroShift services, clean up the custom certificates and re-create the custom certificates, use the following steps.

Procedure

  1. Stop the MicroShift services and clean up the custom certificates by running the following command: 

    $ sudo microshift-cleanup-data --cert
    Copy to Clipboard Toggle word wrap

    Example output

    Stopping MicroShift services
Removing MicroShift certificates
MicroShift service was stopped
Cleanup succeeded
    Copy to Clipboard Toggle word wrap

  2. Restart the MicroShift services to recreate the custom certificates by running the following command: 

    $ sudo systemctl start microshift
    Copy to Clipboard Toggle word wrap

9.1.6. Additional resources
Copy link

9.2. Configuring TLS security profiles
Copy link

Use transport layer security (TLS) protocols to help prevent known insecure protocols, ciphers, or algorithms from accessing the applications you run on MicroShift.

9.2.1. Using TLS with MicroShift
Copy link

Transport layer security (TLS) profiles provide a way for servers to regulate which ciphers a client can use when connecting to the server. Using TLS helps to ensure that MicroShift applications use cryptographic libraries that do not allow known insecure protocols, ciphers, or algorithms. You can use either the TLS 1.2 or TLS 1.3 security profiles with MicroShift.

MicroShift API server cipher suites apply automatically to the following internal control plane components:

  • API server
  • Kubelet
  • Kube controller manager
  • Kube scheduler
  • etcd
  • Route controller manager

The API server uses the configured minimum TLS version and the associated cipher suites. If you leave the cipher suites parameter empty, the defaults for the configured minimum version are used automatically.

Default cipher suites for TLS 1.2

The following list specifies the default cipher suites for TLS 1.2:

  • TLS_AES_128_GCM_SHA256
  • TLS_AES_256_GCM_SHA384
  • TLS_CHACHA20_POLY1305_SHA256
  • TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  • TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  • TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  • TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  • TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
  • TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256

Default cipher suites for TLS 1.3

The following list specifies the default cipher suites for TLS 1.3:

  • TLS_AES_128_GCM_SHA256
  • TLS_AES_256_GCM_SHA384
  • TLS_CHACHA20_POLY1305_SHA256

9.2.2. Configuring TLS for MicroShift
Copy link

You can choose to use either the TLS 1.2 or TLS 1.3 security profiles with MicroShift for system hardening.

Prerequisites

  • You have access to the node as a root user.
  • MicroShift has either not started for the first time, or is stopped.
  • The OpenShift CLI (oc) is installed.
  • The certificate authority has issued the custom certificates (CAs).

Procedure

  1. Make a copy of the provided config.yaml.default file in the /etc/microshift/ directory, renaming it config.yaml.

  2. Keep the new MicroShift config.yaml in the /etc/microshift/ directory. Your config.yaml file is read every time the MicroShift service starts.

    Note

    After you create it, the config.yaml file takes precedence over built-in settings.

  3. Optional: Use a configuration snippet if you are using an existing MicroShift YAML. See "Using configuration snippets" in the Additional resources section for more information.

  4. Replace the default values in the tls section of the MicroShift YAML with your valid values.

    Example TLS 1.2 configuration

    apiServer:
# ...
  tls:
    cipherSuites:
    1 
    
    - <cipher_suite_1>
    2 
    
    - ...
    minVersion: VersionTLS12
    3 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    Defaults to the suites of the configured minVersion. If minVersion is not configured, the default value is TLS 1.2.
    2
    Specify the cipher suites you want to use from the list of supported cipher suites. If you do not configure this list, all of the supported cipher suites are used. All clients connecting to the API server must support the configured cipher suites or the connections fail during the TLS handshake phase. Be sure to add the CA certificate bundle to the list of CA certificates that the TLS client or server trusts.
    3
    Specify VersionTLS12 or VersionTLS13.
    Important

    When you choose TLS 1.3 as the minimum TLS version, only the default MicroShift cipher suites can be used. Additional cipher suites are not configurable. If other cipher suites to use with TLS 1.3 are configured, those suites are ignored and overwritten by the MicroShift defaults.

  5. Complete any other additional configurations that you require, then restart MicroShift by running the following command: 

    $ sudo systemctl restart microshift
    Copy to Clipboard Toggle word wrap
9.2.2.1. Default cipher suites
Copy link

Default cipher suites are included with MicroShift for both TLS 1.2 and TLS 1.3. The cipher suites for TLS 1.3 cannot be customized.

Default cipher suites for TLS 1.2

The following list specifies the default cipher suites for TLS 1.2:

  • TLS_AES_128_GCM_SHA256
  • TLS_AES_256_GCM_SHA384
  • TLS_CHACHA20_POLY1305_SHA256
  • TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  • TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  • TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  • TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  • TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
  • TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256

Default cipher suites for TLS 1.3

The following list specifies the default cipher suites for TLS 1.3:

  • TLS_AES_128_GCM_SHA256
  • TLS_AES_256_GCM_SHA384
  • TLS_CHACHA20_POLY1305_SHA256

9.3. Configuring audit logging policies
Copy link

You can control MicroShift audit log file rotation and retention by using configuration values.

9.3.1. About setting limits on audit log files
Copy link

Controlling the rotation and retention of the MicroShift audit log file by using configuration values helps keep the limited storage capacities of far-edge devices from being exceeded. On such devices, logging data accumulation can limit host system or node workloads, potentially causing the device stop working. Setting audit log policies can help ensure that critical processing space is continually available.

The values you set to limit MicroShift audit logs enable you to enforce the size, number, and age limits of audit log backups. Field values are processed independently of one another and without prioritization.

You can set fields in combination to define a maximum storage limit for retained logs. For example:

  • Set both maxFileSize and maxFiles to create a log storage upper limit.
  • Set a maxFileAge value to automatically delete files older than the timestamp in the file name, regardless of the maxFiles value.
9.3.1.1. Default audit log values
Copy link

MicroShift includes the following default audit log rotation values:

Expand
Table 9.2. MicroShift default audit log values
Audit log parameterDefault settingDefinition

maxFileAge:

0

How long log files are retained before automatic deletion. The default value means that a log file is never deleted based on age. This value can be configured.

maxFiles:

10

The total number of log files retained. By default, MicroShift retains 10 log files. The oldest is deleted when an excess file is created. This value can be configured.

maxFileSize:

200

By default, when the audit.log file reaches the maxFileSize limit, the audit.log file is rotated and MicroShift begins writing to a new audit.log file. This value is in megabytes and can be configured.

profile:

Default

The Default profile setting only logs metadata for read and write requests; request bodies are not logged except for OAuth access token requests. If you do not specify this field, the Default profile is used.

The maximum default storage usage for audit log retention is 2000Mb if there are 10 or fewer files.

If you do not specify a value for a field, the default value is used. If you remove a previously set field value, the default value is restored after the next MicroShift service restart.

Important

You must configure audit log retention and rotation in Red Hat Enterprise Linux (RHEL) for logs that are generated by application pods. These logs print to the console and are saved. Ensure that your log preferences are configured for the RHEL /var/log/audit/audit.log file to maintain MicroShift node health.

9.3.2. About audit log policy profiles
Copy link

Audit log profiles define how to log requests that come to the OpenShift API server and the Kubernetes API server.

MicroShift supports the following predefined audit policy profiles:

Expand
ProfileDescription

Default

Logs only metadata for read and write requests; does not log request bodies except for OAuth access token requests. This is the default policy.

WriteRequestBodies

In addition to logging metadata for all requests, logs request bodies for every write request to the API servers (create, update, patch, delete, deletecollection). This profile has more resource overhead than the Default profile. [1]

AllRequestBodies

In addition to logging metadata for all requests, logs request bodies for every read and write request to the API servers (get, list, create, update, patch). This profile has the most resource overhead. [1]

None

No requests are logged, including OAuth access token requests and OAuth authorize token requests.

Warning

Do not disable audit logging by using the None profile unless you are fully aware of the risks of not logging data that can be beneficial when troubleshooting issues. If you disable audit logging and a support situation arises, you might need to enable audit logging and reproduce the issue to troubleshoot properly.

  1. Sensitive resources, such as Secret, Route, and OAuthClient objects, are only logged at the metadata level.

By default, MicroShift uses the Default audit log profile. You can use another audit policy profile that also logs request bodies, but be aware of the increased resource usage such as CPU, memory, and I/O.

9.3.3. Configuring audit log values
Copy link

You can configure audit log settings by using the MicroShift service configuration file.

Procedure

  1. Make a copy of the provided config.yaml.default file in the /etc/microshift/ directory, renaming it config.yaml. Keep the new MicroShift config.yaml you create in the /etc/microshift/ directory. The new config.yaml is read whenever the MicroShift service starts. After you create it, the config.yaml file takes precedence over built-in settings.

  2. Replace the default values in the auditLog section of the YAML with your desired valid values.

    Example default auditLog configuration

    apiServer:
# ....
  auditLog:
    maxFileAge: 7
    1 
    
    maxFileSize: 200
    2 
    
    maxFiles: 1
    3 
    
    profile: Default
    4 
    
# ....
    Copy to Clipboard Toggle word wrap

    1
    Specifies the maximum time in days that log files are kept. Files older than this limit are deleted. In this example, after a log file is more than 7 days old, it is deleted. The files are deleted regardless of whether or not the live log has reached the maximum file size specified in the maxFileSize field. File age is determined by the timestamp written in the name of the rotated log file, for example, audit-2024-05-16T17-03-59.994.log. When the value is 0, the limit is disabled.
    2
    The maximum audit log file size in megabytes. In this example, the file is rotated as soon as the live log reaches the 200 MB limit. When the value is set to 0, the limit is disabled.
    3
    The maximum number of rotated audit log files retained. After the limit is reached, the log files are deleted in order from oldest to newest. In this example, the value 1 results in only 1 file of size maxFileSize being retained in addition to the current active log. When the value is set to 0, the limit is disabled.
    4
    Logs only metadata for read and write requests; does not log request bodies except for OAuth access token requests. If you do not specify this field, the Default profile is used.

  3. Optional: To specify a new directory for logs, you can stop MicroShift, and then move the /var/log/kube-apiserver directory to your desired location:

    1. Stop MicroShift by running the following command: 

      $ sudo systemctl stop microshift
      Copy to Clipboard Toggle word wrap

    2. Move the /var/log/kube-apiserver directory to your desired location by running the following command: 

      $ sudo mv /var/log/kube-apiserver <~/kube-apiserver>
      1
      Copy to Clipboard Toggle word wrap
      1
      Replace <~/kube-apiserver> with the path to the directory that you want to use.

    3. If you specified a new directory for logs, create a symlink to your custom directory at /var/log/kube-apiserver by running the following command: 

      $ sudo ln -s <~/kube-apiserver> /var/log/kube-apiserver
      1
      Copy to Clipboard Toggle word wrap
      1
      Replace <~/kube-apiserver> with the path to the directory that you want to use. This enables the collection of logs in sos reports.

  4. If you are configuring audit log policies on a running instance, restart MicroShift by entering the following command: 

    $ sudo systemctl restart microshift
    Copy to Clipboard Toggle word wrap

9.3.4. Troubleshooting audit log configuration
Copy link

Use the following steps to troubleshoot custom audit log settings and file locations.

Procedure

  • Check the current values that are configured by running the following command: 

    $ sudo microshift show-config --mode effective
    Copy to Clipboard Toggle word wrap

    Example output

    auditLog:
    maxFileSize: 200
    maxFiles: 1
    maxFileAge: 7
    profile: AllRequestBodies
    Copy to Clipboard Toggle word wrap

  • Check the audit.log file permissions by running the following command: 

    $ sudo ls -ltrh /var/log/kube-apiserver/audit.log
    Copy to Clipboard Toggle word wrap

    Example output

    -rw-------. 1 root root 46M Mar 12 09:52 /var/log/kube-apiserver/audit.log
    Copy to Clipboard Toggle word wrap

  • List the contents of the current log directory by running the following command: 

    $ sudo ls -ltrh /var/log/kube-apiserver/
    Copy to Clipboard Toggle word wrap

    Example output

    total 6.0M
-rw-------. 1 root root 2.0M Mar 12 10:56 audit-2024-03-12T14-56-16.267.log
-rw-------. 1 root root 2.0M Mar 12 10:56 audit-2024-03-12T14-56-49.444.log
-rw-------. 1 root root 962K Mar 12 10:57 audit.log
    Copy to Clipboard Toggle word wrap

You can enhance supply chain security by using the sigstore signing methodology.

Important

sigstore support is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

You can configure the container runtime to verify image integrity by using the sigstore signing methodology. Configuring MicroShift container runtimes enables the verification of image integrity. With the sigstore project, developers can digitally sign what they build, creating a safer chain of custody that traces software back to the source. Administrators can then verify signatures and monitor workflows at scale. By using sigstore, you can store signatures in the same registry as the build images.

  • For user-specific images, you must update the configuration file to point to the appropriate public key, or disable signature verification for those image sources.
Important

For disconnected or offline configurations, you must embed the public key contents into the operating system image.

Verify container signatures for MicroShift by configuring the container runtime to use sigstore. The container signature verification uses the public key from the Red Hat keypair when signing the images. To use sigstore, edit the default /etc/containers/policy.json file that is installed as part of the container runtime package.

You can access Red Hat public keys at the following link:

You must use the release key 3 for verifying MicroShift container signatures.

Prerequisites

  • You have admin access to the MicroShift host.
  • You installed MicroShift.

Procedure

  1. Download the relevant public key and save it as /etc/containers/RedHat_ReleaseKey3.pub by running the following command: 

    $ sudo curl -sL https://access.redhat.com/security/data/63405576.txt -o /etc/containers/RedHat_ReleaseKey3.pub
    Copy to Clipboard Toggle word wrap

  2. To configure the container runtime to verify images from Red Hat sources, edit the /etc/containers/policy.json file to contain the following configuration:

    Example policy JSON file

    {
    "default": [
        {
            "type": "reject"
        }
    ],
    "transports": {
        "docker": {
            "quay.io/openshift-release-dev": [{
                "type": "sigstoreSigned",
                "keyPath": "/etc/containers/RedHat_ReleaseKey3.pub",
                "signedIdentity": {
                    "type": "matchRepoDigestOrExact"
                }
            }],
            "registry.redhat.io": [{
                "type": "sigstoreSigned",
                "keyPath": "/etc/containers/RedHat_ReleaseKey3.pub",
                "signedIdentity": {
                    "type": "matchRepoDigestOrExact"
                }
            }]
        }
    }
}
    Copy to Clipboard Toggle word wrap

  3. Configure Red Hat remote registries to use sigstore attachments when pulling images to the local storage, by editing the /etc/containers/registries.d/registry.redhat.io.yaml file to contain the following configuration: 

    $ cat /etc/containers/registries.d/registry.redhat.io.yaml
docker:
     registry.redhat.io:
         use-sigstore-attachments: true
    Copy to Clipboard Toggle word wrap

  4. Configure Red Hat remote registries to use sigstore attachments when pulling images to the local storage, by editing the /etc/containers/registries.d/registry.quay.io.yaml file to contain the following configuration: 

    $ cat /etc/containers/registries.d/quay.io.yaml
docker:
  quay.io/openshift-release-dev:
    use-sigstore-attachments: true
    Copy to Clipboard Toggle word wrap
  5. Create user-specific registry configuration files if your use case requires signature verification for those image sources. You can use the example here to start with and add your own requirements.

Next steps

  1. If you are using a mirror registry, enable sigstore attachments.
  2. Otherwise, proceed to wiping the local container storage clean.

If you are using mirror registries you must apply additional configuration to enable sigstore attachments and mirroring by digest.

Prerequisites

  • You have admin access to the MicroShift host.
  • You completed the steps in "Verifying container signatures using sigstore."

Procedure

  1. Enable sigstore attachments by creating the /etc/containers/registries.d/mirror.registry.local.yaml file. 

    $ cat /etc/containers/registries.d/<mirror.registry.local.yaml>
    1 
    
docker:
   mirror.registry.local:
        use-sigstore-attachments: true
    Copy to Clipboard Toggle word wrap
    1
    Name the <mirror.registry.local.yaml> file after your mirror registry URL.

  2. Enable mirroring by digest by creating the /etc/containers/registries.conf.d/999-microshift-mirror.conf with the following contents: 

    $ cat /etc/containers/registries.conf.d/999-microshift-mirror.conf
[[registry]]
    prefix = "quay.io/openshift-release-dev"
    location = "mirror.registry.local"
    mirror-by-digest-only = true

[[registry]]
    prefix = "registry.redhat.io"
    location = "mirror.registry.local"
    mirror-by-digest-only = true
    Copy to Clipboard Toggle word wrap

Next steps

  1. Wipe the local container storage clean.
9.4.2.2. Wiping local container storage clean
Copy link

When you apply the configuration to an existing system, you must wipe the local container storage clean. Cleaning the container storage ensures that container images with signatures are properly downloaded.

Prerequisites

  • You have administrator access to the MicroShift host.
  • You enabled sigstore on your mirror registries.

Procedure

  1. Stop the CRI-O container runtime service and MicroShift by running the following command: 

    $ sudo systemctl stop crio microshift
    Copy to Clipboard Toggle word wrap

  2. Wipe the CRI-O container runtime storage clean by running the following command: 

    $ sudo crio wipe --force
    Copy to Clipboard Toggle word wrap

  3. Restart the CRI-O container runtime service and MicroShift by running the following command: 

    $ sudo systemctl start crio microshift
    Copy to Clipboard Toggle word wrap

Verification

Verify that all pods are running in a healthy state by entering the following command: 

$ oc get pods -A
Copy to Clipboard Toggle word wrap

Example output

NAMESPACE                   NAME                                                     READY   STATUS   RESTARTS  AGE
default                     i-06166fbb376f14a8bus-west-2computeinternal-debug-qtwcr  1/1     Running  0		    46m
kube-system                 csi-snapshot-controller-5c6586d546-lprv4                 1/1     Running  0		    51m
openshift-dns               dns-default-45jl7                                        2/2     Running  0		    50m
openshift-dns               node-resolver-7wmzf                                      1/1     Running  0		    51m
openshift-ingress           router-default-78b86fbf9d-qvj9s                          1/1     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-master-5rfhh                                     4/4     Running  0		    51m
openshift-ovn-kubernetes    ovnkube-node-gcnt6                                       1/1     Running  0		    51m
openshift-service-ca        service-ca-bf5b7c9f8-pn6rk                               1/1     Running  0		    51m
openshift-storage           topolvm-controller-549f7fbdd5-7vrmv                      5/5     Running  0		    51m
openshift-storage           topolvm-node-rht2m                                       3/3     Running  0		    50m
Copy to Clipboard Toggle word wrap

Note

This example output shows basic MicroShift. If you have installed optional RPMs, the status of pods running those services is also expected to be shown in your output.

Chapter 10. Configuring low latency
Copy link

10.1. Configuring low latency
Copy link

You can configure and tune low latency capabilities to improve application performance on edge devices.

Latency is defined as the time from an event to the response to that event. You can use low latency configurations and tuning in a MicroShift node running in an operational or software-defined control system where an edge device has to respond quickly to an external event. You can fully optimize low latency performance by combining MicroShift configurations with operating system tuning and workload partitioning.

Important

The CPU set for management applications, such as the MicroShift service, OVS, CRI-O, MicroShift pods, and isolated cores, must contain all-online CPUs.

To configure low latency for applications running in a MicroShift node, you must complete the following tasks:

Required
  • Install the microshift-low-latency RPM.
  • Configure workload partitioning.
  • Configure the kubelet section of the config.yaml file in the /etc/microshift/ directory.
  • Configure and activate a TuneD profile. TuneD is a Red Hat Enterprise Linux (RHEL) service that monitors the host system and optimizes performance under certain workloads.
  • Restart the host.
Optional

When you install MicroShift, the low latency RPM package is not installed by default. You can install the low latency RPM as an optional package.

Prerequisites

  1. You installed the MicroShift RPM.
  2. You configured workload partitioning for MicroShift.

Procedure

  • Install the low latency RPM package by running the following command: 

    $ sudo dnf install -y microshift-low-latency
    Copy to Clipboard Toggle word wrap
    Tip

    Wait to restart the host until after activating your TuneD profile. Restarting the host restarts MicroShift and CRI-O, which applies the low latency manifests and activates the TuneD profile.

Next steps

  1. Configure the kubelet parameter for low latency in the MicroShift config.yaml.
  2. Tune your operating system, for example, configure and activate a TuneD profile.
  3. Optional: Configure automatic activation of your TuneD profile.
  4. Optional: If you are using the x86_64 architecture, install Red Hat Enterprise Linux for Real Time (real-time kernel).
  5. Prepare your workloads for low latency.

The first step in enabling low latency to a MicroShift node is to add configurations to the MicroShift config.yaml file.

Prerequisites

  • You installed the OpenShift CLI (oc).
  • You have root access to the node.
  • You made a copy of the provided config.yaml.default file in the /etc/microshift/ directory, and renamed it config.yaml.

Procedure

  • Add the kubelet configuration to the MicroShift config.yaml file:

    Example passthrough kubelet configuration

    apiServer:
# ...
kubelet:
    1 
    
  cpuManagerPolicy: static
    2 
    
  cpuManagerPolicyOptions:
    full-pcpus-only: "true"
    3 
    
  cpuManagerReconcilePeriod: 5s
  memoryManagerPolicy: Static
    4 
    
  topologyManagerPolicy: single-numa-node
  reservedSystemCPUs: 0-1
    5 
    
  reservedMemory:
  - limits:
      memory: 1100Mi
    6 
    
    numaNode: 0
  kubeReserved:
    memory: 500Mi
  systemReserved:
    memory: 500Mi
  evictionHard:
    7 
    
    imagefs.available: "15%"
    8 
    
    memory.available: "100Mi"
    9 
    
    nodefs.available: "10%"
    10 
    
    nodefs.inodesFree: "5%"
    11 
    
  evictionPressureTransitionPeriod: 5m
    12 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    If you change the CPU or memory managers in the kubelet configuration, you must remove files that cache the previous configuration. Restart the host to remove them automatically, or manually remove the /var/lib/kubelet/cpu_manager_state and /var/lib/kubelet/memory_manager_state files.
    2
    The name of the policy to use. Valid values are none and static. Requires the CPUManager feature gate to be enabled. Default value is none.
    3
    A set of key=value pairs for setting extra options that fine tune the behavior of the CPUManager policies. The default value is null. Requires both the CPUManager and CPUManagerPolicyOptions feature gates to be enabled.
    4
    The name of the policy used by Memory Manager. Case-sensitive. The default value is none. Requires the MemoryManager feature gate to be enabled.
    5
    Required. The reservedSystemCPUs value must be the inverse of the offlined CPUs because both values combined must account for all of the CPUs on the system. This parameter is essential to dividing the management and application workloads. Use this parameter to define a static CPU set for the host-level system and Kubernetes daemons, plus interrupts and timers. Then the rest of the CPUs on the system can be used exclusively for workloads.
    6
    The value in reservedMemory[0].limits.memory, 1100 Mi in this example, is equal to kubeReserved.memory + systemReserved.memory + evictionHard.memory.available.
    7
    The evictionHard parameters define under which conditions the kubelet evicts pods. When you change the default value of only one parameter for the evictionHard stanza, the default values of other parameters are not inherited and are set to zero. Provide all the threshold values even when you want to change just one.
    8
    The imagefs is a filesystem that container runtimes use to store container images and container writable layers. In this example, the evictionHard.imagefs.available parameter means that the pod is evicted when the available space of the image filesystem is less than 15%.
    9
    In this example, the evictionHard.memory.available parameter means that the pods are evicted when the available memory of the node drops below 100MiB.
    10
    In this example, the evictionHard.nodefs.available parameter means that the pods are evicted when the main filesystem of the node has less than 10% available space.
    11
    In this example, the evictionHard.nodefs.inodesFree parameter means that the pods are evicted when more than 15% of the node’s main filesystem’s inodes are in use.
    12
    For container garbage collection: The duration to wait before transitioning out of an eviction pressure condition. Setting the evictionPressureTransitionPeriod parameter to 0 configures the default value of 5 minutes.

Verification

  • After you complete the next steps and restart the host, you can use a root-access account to check that your settings are in the config.yaml file in the /var/lib/microshift/resources/kubelet/config/ directory.

Next steps

  1. Enable workload partitioning.
  2. Tune your operating system. For example, configure and activate a TuneD profile.
  3. Optional: Configure automatic enablement of your TuneD profile.
  4. Optional: If you are using the x86_64 architecture, you can install Red Hat Enterprise Linux for Real Time (real-time kernel).
  5. Prepare your MicroShift workloads for low latency.

10.1.4. Tuning Red Hat Enterprise Linux 9
Copy link

As a Red Hat Enterprise Linux (RHEL) system administrator, you can use the TuneD service to optimize the performance profile of RHEL for a variety of use cases. TuneD monitors and optimizes system performance under certain workloads, including latency performance.

  • Use TuneD profiles to tune your system for different use cases, such as deploying a low-latency MicroShift node.
  • You can modify the rules defined for each profile and customize tuning for a specific device.
  • When you switch to another profile or deactivate TuneD, all changes made to the system settings by the previous profile revert back to their original state.
  • You can also configure TuneD to react to changes in device usage and adjusts settings to improve performance of active devices and reduce power consumption of inactive devices.
10.1.4.1. Configuring the MicroShift TuneD profile
Copy link

Configure a TuneD profile for your host to use low latency with MicroShift workloads by using the microshift-baseline-variables.conf configuration file provided in the Red Hat Enterprise Linux (RHEL) /etc/tuned/ host directory after you install the microshift-low-latency RPM package.

Prerequisites

  • You have root access to the node.
  • You installed the microshift-low-latency RPM package.
  • Your RHEL host has TuneD installed. See Getting started with TuneD (RHEL documentation).

Procedure

  1. You can use the default microshift-baseline-variables.conf TuneD profile in the /etc/tuned/ directory profile, or create your own to add more tunings.

    Example microshift-baseline-variables.conf TuneD profile

    # Isolate cores 2-7 for running application workloads
isolated_cores=2-7
    1 
    

# Size of the hugepages
hugepages_size=2M
    2 
    

# Number of hugepages
hugepages=0

# Additional kernel arguments
additional_args=
    3 
    

# CPU set to be offlined
offline_cpu_set=
    4
    Copy to Clipboard Toggle word wrap

    1
    Controls which cores should be isolated. By default, 1 core per socket is reserved in MicroShift for housekeeping. The other cores are isolated. Valid values are a core list or range. You can isolate any range, for example: isolated_cores=2,4-7 or isolated_cores=2-23.
    Important

    You must keep only one isolated_cores= variable.

    Note

    The Kubernetes CPU manager can use any CPU to run the workload except the reserved CPUs defined in the kubelet configuration. For this reason it is best that:

    • The sum of the kubelet’s reserved CPUs and isolated cores include all online CPUs.
    • Isolated cores are complementary to the reserved CPUs defined in the kubelet configuration.
    2
    Size of the hugepages. Valid values are 2M or 1G.
    3
    Additional kernel arguments, for example, additional_args=console=tty0 console=ttyS0,115200.
    4
    The CPU set to be offlined.
    Important

    Must not overlap with isolated_cores.

  2. Enable the profile or make changes active, by running the following command: 

    $ sudo tuned-adm profile microshift-baseline
    Copy to Clipboard Toggle word wrap
  3. Reboot the host to make kernel arguments active.

Verification

  • Optional: You can read the /proc/cmdline file that contains the arguments given to the currently running kernel on start. 

    $ cat /proc/cmdline
    Copy to Clipboard Toggle word wrap

    Example output

    BOOT_IMAGE=(hd0,msdos2)/ostree/rhel-7f82ccd9595c3c70af16525470e32c6a81c9138c4eae6c79ab86d5a2d108d7fc/vmlinuz-5.14.0-427.31.1.el9_4.x86_64+rt crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.lvm.lv=rhel/root fips=0 console=ttyS0,115200n8 root=/dev/mapper/rhel-root rw ostree=/ostree/boot.1/rhel/7f82ccd9595c3c70af16525470e32c6a81c9138c4eae6c79ab86d5a2d108d7fc/0 skew_tick=1 tsc=reliable rcupdate.rcu_normal_after_boot=1 nohz=on nohz_full=2,4-5 rcu_nocbs=2,4-5 tuned.non_isolcpus=0000000b intel_pstate=disable nosoftlockup hugepagesz=2M hugepages=10
    Copy to Clipboard Toggle word wrap

Next steps

  1. Prepare your MicroShift workloads for low latency.
  2. Optional: Configure automatic enablement of your TuneD profile.
  3. Optional: If you are using the x86_64 architecture, you can install Red Hat Enterprise Linux for Real Time (real-time kernel).

Included in the microshift-low-latency RPM package is a systemd service that you can configure to automatically enable a TuneD profile when the system starts. This ability is particularly useful if you are installing MicroShift in a large fleet of devices.

Prerequisites

  1. You installed the microshift-low-latency RPM package on the host.
  2. You enabled low latency in the MicroShift config.yaml.
  3. You created a TuneD profile.
  4. You configured the microshift-baseline-variables.conf file.

Procedure

  1. Configure the tuned.yaml in the /etc/microshift/ directory, for example:

    Example tuned.yaml

    profile: microshift-baseline
    1 
    
reboot_after_apply: True
    2
    Copy to Clipboard Toggle word wrap

    1
    Controls which TuneD profile is activated. In this example, the name of the profile is microshift-baseline.
    2
    Controls whether the host must be rebooted after applying the profile. Valid values are True and False. For example, use the True setting to automatically restart the host after a new ostree commit is deployed.
    Important

    The host is restarted when the microshift-tuned.service runs, but it does not restart the system when a new commit is deployed. You must restart the host to enable a new commit, then the system starts again when the microshift-tuned.service runs on that boot and detects changes to profiles and variables.

    This double-boot can effect rollbacks. Ensure that you adjust the number of reboots in greenboot that are allowed before rollback when using automatic profile activation. For example, if 3 reboots are allowed before a rollback in greenboot, increase that number to 4. See the "Additional resources" list for more information.

  2. Enable the microshift-tuned.service to run on each system start by entering the following command: 

    $ sudo systemctl enable microshift-tuned.service
    Copy to Clipboard Toggle word wrap
    Important

    If you set reboot_after_apply to True, ensure that a TuneD profile is active and that no other profiles have been activated outside of the MicroShift service. Otherwise, starting the microshift-tuned.service results in a host reboot.

  3. Start the microshift-tuned.service by running the following command: 

    $ sudo systemctl start microshift-tuned.service
    Copy to Clipboard Toggle word wrap
    Note

    The microshift-tuned.service uses collected checksums to detect changes to selected TuneD profiles and variables. If there are no checksums on the disk, the service activates the TuneD profile and restarts the host. Expect a host restart when first starting the microshift-tuned.service.

Next steps

  • Optional: If you are using the x86_64 architecture, you can install Red Hat Enterprise Linux for Real Time (real-time kernel).

If your workload has stringent low-latency determinism requirements for core kernel features such as interrupt handling and process scheduling in the microsecond (μs) range, you can use the Red Hat Enterprise Linux for Real Time (real-time kernel). The goal of the real-time kernel is consistent, low-latency determinism that offers predictable response times.

When considering system tuning, consider the following factors:

  • System tuning is just as important when using the real-time kernel as it is for the standard kernel.
  • Installing the real-time kernel on an untuned system running the standard kernel supplied as part of the RHEL 9 release is not likely to result in any noticeable benefit.
  • Tuning the standard kernel yields 90% of possible latency gains.
  • The real-time kernel provides the last 10% of latency reduction required by the most demanding workloads.

Although the real-time kernel is not necessary for low latency workloads, using the real-time kernel can optimize low latency performance. You can install it on a host by using RPM packages, and include it in a Red Hat Enterprise Linux for Edge (RHEL for Edge) image deployment.

Prerequisites

  • You have a Red Hat subscription that includes Red Hat Enterprise Linux for Real Time (real-time kernel). For example, your host machine is registered and Red Hat Enterprise Linux (RHEL) is attached to a RHEL for Real Time subscription.
  • You are using x86_64 architecture.

Procedure

  1. Enable the real-time kernel repository by running the following command: 

    $ sudo subscription-manager repos --enable rhel-9-for-x86_64-rt-rpms
    Copy to Clipboard Toggle word wrap

  2. Install the real-time kernel by running the following command: 

    $ sudo dnf install -y kernel-rt
    Copy to Clipboard Toggle word wrap

  3. Query the real-time kernel version by running the following command: 

    $ RTVER=$(rpm -q --queryformat '%{version}-%{release}.%{arch}' kernel-rt | sort | tail -1)
    Copy to Clipboard Toggle word wrap

  4. Make a persistent change in GRUB that designates the real-time kernel as the default kernel by running the following command: 

    $ sudo grubby --set-default="/boot/vmlinuz-${RTVER}+rt"
    Copy to Clipboard Toggle word wrap
  5. Restart the host to activate the real-time kernel.

Next steps

  1. Prepare your MicroShift workloads for low latency.
  2. Optional: Use a blueprint to install the real-time kernel in a RHEL for Edge image.

You can include the real-time kernel in a RHEL for Edge image deployment using image builder. The following example blueprint sections include references gathered from the previous steps required to configure low latency for a MicroShift node.

Prerequisites

  • You have a Red Hat subscription enabled on the host that includes Red Hat Enterprise Linux for Real Time (real-time kernel).
  • You are using the x86_64 architecture.
  • You configured osbuild to use the kernel-rt repository.
Important

A subscription that includes the real-time kernel must be enabled on the host used to build the commit.

Procedure

  • Add the following example blueprint sections to your complete installation blueprint for installing the real-time kernel in a RHEL for Edge image:

    Example blueprint snippet for the real-time kernel

    [[packages]]
name = "microshift-low-latency"
version = "*"

# Kernel RT is supported only on the x86_64 architecture
[customizations.kernel]
name = "kernel-rt"

[customizations.services]
enabled = ["microshift", "microshift-tuned"]

[[customizations.files]]
path = "/etc/microshift/config.yaml"
data = """
kubelet:
  cpuManagerPolicy: static
  cpuManagerPolicyOptions:
    full-pcpus-only: "true"
  cpuManagerReconcilePeriod: 5s
  memoryManagerPolicy: Static
  topologyManagerPolicy: single-numa-node
  reservedSystemCPUs: 0-1
  reservedMemory:
  - limits:
      memory: 1100Mi
    numaNode: 0
  kubeReserved:
    memory: 500Mi
  systemReserved:
    memory: 500Mi
  evictionHard:
    imagefs.available: 15%
    memory.available: 100Mi
    nodefs.available: 10%
    nodefs.inodesFree: 5%
  evictionPressureTransitionPeriod: 5m
"""

[[customizations.files]]
path = "/etc/tuned/microshift-baseline-variables.conf"
data = """
# Isolated cores should be complementary to the kubelet configuration reserved CPUs.
# Isolated and reserved CPUs must contain all online CPUs.
# Core #3 is for testing offlining, therefore it is skipped.
isolated_cores=2,4-5
hugepages_size=2M
hugepages=10
additional_args=test1=on test2=true dummy
offline_cpu_set=3
"""

[[customizations.files]]
path = "/etc/microshift/tuned.yaml"
data = """
profile: microshift-baseline
reboot_after_apply: True
"""
    Copy to Clipboard Toggle word wrap

Next steps

  1. Complete the image building process.
  2. If you have not completed the previous steps for enabling low latency for your MicroShift cluster, do so now. Update the blueprint with the information gathered in those steps.
  3. If you have not configured workload partitioning, do so now.
  4. Prepare your MicroShift workloads for low latency.

Complete the build process by starting with the following procedure to embed MicroShiftin a RHEL for Edge image. Then complete the remaining steps in the installation documentation for installing MicroShift in a RHEL for Edge image:

To take advantage of low latency, workloads running on MicroShift must have the microshift-low-latency container runtime configuration set by using the RuntimeClass feature. The CRI-O RuntimeClass object is installed with the microshift-low-latency RPM, so only the pod annotations need to be configured.

Prerequisites

  • You installed the microshift-low-latency RPM package.
  • You configured workload partitioning.

Procedure

  • Use the following example to set the following annotations in the pod spec: 

    cpu-load-balancing.crio.io: "disable"
irq-load-balancing.crio.io: "disable"
cpu-quota.crio.io: "disable"
cpu-load-balancing.crio.io: "disable"
cpu-freq-governor.crio.io: "<governor>"
    Copy to Clipboard Toggle word wrap

    Example pod that runs oslat test:

    apiVersion: v1
kind: Pod
metadata:
  name: oslat
  annotations:
    cpu-load-balancing.crio.io: "disable"
    1 
    
    irq-load-balancing.crio.io: "disable"
    2 
    
    cpu-quota.crio.io: "disable"
    3 
    
    cpu-c-states.crio.io: "disable"
    4 
    
    cpu-freq-governor.crio.io: "<governor>"
    5 
    
spec:
  runtimeClassName: microshift-low-latency
    6 
    
  containers:
  - name: oslat
    image: quay.io/container-perf-tools/oslat
    imagePullPolicy: Always
    resources:
      requests:
        memory: "400Mi"
        cpu: "2"
      limits:
        memory: "400Mi"
        cpu: "2"
    env:
    - name: tool
      value: "oslat"
    - name: manual
      value: "n"
    - name: PRIO
      value: "1"
    - name: delay
      value: "0"
    - name: RUNTIME_SECONDS
      value: "60"
    - name: TRACE_THRESHOLD
      value: ""
    - name: EXTRA_ARGS
      value: ""
    securityContext:
      privileged: true
      capabilities:
        add:
          - SYS_NICE
          - IPC_LOCK
    Copy to Clipboard Toggle word wrap

    1
    Disables the CPU load balancing for the pod.
    2
    Opts the pod out of interrupt handling (IRQ).
    3
    Disables the CPU completely fair scheduler (CFS) quota at the pod run time.
    4
    Enables or disables C-states for each CPU. Set the value to disable to provide the best performance for a high-priority pod.
    5
    Sets the cpufreq governor for each CPU. The performance governor is recommended for high-priority workloads.
    6
    The runtimeClassName must match the name of the performance profile configured in the node. For example, microshift-low-latency.
    Note

    Disable CPU load balancing only when the CPU manager static policy is enabled and for pods with guaranteed QoS that use whole CPUs. Otherwise, disabling CPU load balancing can affect the performance of other containers in the node.

    Important

    For the pod to have the Guaranteed QoS class, it must have the same values of CPU and memory in requests and limits. See Guaranteed (Kubernetes upstream documentation)

An image blueprint is a persistent definition of the required image customizations that enable you to create multiple builds. Instead of reconfiguring the blueprint for each image build, you can edit, rebuild, delete, and save the blueprint so that you can keep rebuilding images from it.

Example blueprint used to install the real-time kernel in a RHEL for Edge image

name = "microshift-low-latency"
description = "RHEL 9.4 and MicroShift configured for low latency"
version = "0.0.1"
modules = []
groups = []
distro = "rhel-94"

[[packages]]
name = "microshift"
version = "*"

[[packages]]
name = "microshift-greenboot"
version = "*"

[[packages]]
name = "microshift-networking"
version = "*"

[[packages]]
name = "microshift-selinux"
version = "*"

[[packages]]
name = "microshift-low-latency"
version = "*"

# Kernel RT is only available for x86_64
[customizations.kernel]
name = "kernel-rt"

[customizations.services]
enabled = ["microshift", "microshift-tuned"]

[customizations.firewall]
ports = ["22:tcp", "80:tcp", "443:tcp", "5353:udp", "6443:tcp", "30000-32767:tcp", "30000-32767:udp"]

[customizations.firewall.services]
enabled = ["mdns", "ssh", "http", "https"]

[[customizations.firewall.zones]]
name = "trusted"
sources = ["10.42.0.0/16", "169.254.169.1"]

[[customizations.files]]
path = "/etc/microshift/config.yaml"
data = """
kubelet:
  cpuManagerPolicy: static
  cpuManagerPolicyOptions:
    full-pcpus-only: "true"
  cpuManagerReconcilePeriod: 5s
  memoryManagerPolicy: Static
  topologyManagerPolicy: single-numa-node
  reservedSystemCPUs: 0-1
  reservedMemory:
  - limits:
      memory: 1100Mi
    numaNode: 0
  kubeReserved:
    memory: 500Mi
  systemReserved:
    memory: 500Mi
  evictionHard:
    imagefs.available: 15%
    memory.available: 100Mi
    nodefs.available: 10%
    nodefs.inodesFree: 5%
  evictionPressureTransitionPeriod: 5m
"""

[[customizations.files]]
path = "/etc/tuned/microshift-baseline-variables.conf"
data = """
# Isolated cores should be complementary to the kubelet configuration reserved CPUs.
# Isolated and reserved CPUs must contain all online CPUs.
# Core #3 is for testing offlining, therefore it is skipped.
isolated_cores=2,4-5
hugepages_size=2M
hugepages=10
additional_args=test1=on test2=true dummy
offline_cpu_set=3
"""

[[customizations.files]]
path = "/etc/microshift/tuned.yaml"
data = """
profile: microshift-baseline
reboot_after_apply: True
"""
Copy to Clipboard Toggle word wrap

10.2. Workload partitioning
Copy link

Workload partitioning divides the node CPU resources into distinct CPU sets. The primary objective is to limit the amount of CPU usage for all control plane components which reserves rest of the device CPU resources for workloads of the user.

Workload partitioning allocates reserved set of CPUs to MicroShift services, cluster management workloads, and infrastructure pods, ensuring that the remaining CPUs in the cluster deployment are untouched and available exclusively for non-platform workloads.

10.2.1. Enabling workload partitioning
Copy link

To enable workload partitioning on MicroShift, make the following configuration changes:

  • Update the MicroShift config.yaml file to include the kubelet configuration file.
  • Create the CRI-O systemd and configuration files.
  • Create and update the systemd configuration file for the MicroShift and CRI-O services respectively.

Procedure

  1. Update the MicroShift config.yaml file to include the kubelet configuration file to enable and configure CPU Manager for the workloads:

    • Create the kubelet configuration file in the path /etc/kubernetes/openshift-workload-pinning. The kubelet configuration directs the kubelet to modify the node resources based on the capacity and allocatable CPUs.

      kubelet configuration example

      # ...
{
  "management": {
    "cpuset": "0,6,7"
      1 
      
  }
}
# ...
      Copy to Clipboard Toggle word wrap

      1
      The cpuset applies to a machine with 8 VCPUs (4 cores) and is valid throughout the document.

    • Update the MicroShift config.yaml file in the path /etc/microshift/config.yaml. Embed the kubelet configuration in the MicroShift config.yaml file to enable and configure CPU Manager for the workloads.

      MicroShift config.yaml example

      # ...
kubelet:
  reservedSystemCPUs: 0,6,7
      1 
      
  cpuManagerPolicy: static
  cpuManagerPolicyOptions:
    full-pcpus-only: "true"
      2 
      
  cpuManagerReconcilePeriod: 5s
# ...
      Copy to Clipboard Toggle word wrap

      1
      Exclusive cpuset for the system daemons and the interrupts/timers.
      2
      kubelet configuration sets the CPUManagerPolicyOptions option to full-pcpus-only to ensure allocation of whole cores to the containers workload.

  2. Create the CRI-O systemd and configuration files:

    • Create the CRI-O configuration file in the path /etc/crio/crio.conf.d/20-microshift-workload-partition.conf which overrides the default configuration that already exists in the 11-microshift-ovn.conf file.

      CRI-O configuration example

      # ...
[crio.runtime]
infra_ctr_cpuset = "0,6,7"

[crio.runtime.workloads.management]
activation_annotation = "target.workload.openshift.io/management"
annotation_prefix = "resources.workload.openshift.io"
resources = { "cpushares" = 0, "cpuset" = "0,6,7" }
# ...
      Copy to Clipboard Toggle word wrap

    • Create the systemd file for CRI-O in the path /etc/systemd/system/crio.service.d/microshift-cpuaffinity.conf.

      CRI-O systemd configuration example

      # ...
[Service]
CPUAffinity=0,6,7
# ...
      Copy to Clipboard Toggle word wrap

  3. Create and update the systemd configuration file with CPUAffinity value for the MicroShift and CRI-O services:

    • Create the MicroShift services systemd file in the path /etc/systemd/system/microshift.service.d/microshift-cpuaffinity.conf. MicroShift will be pinned using the systemd CPUAffinity value.

      MicroShift services systemd configuration example

      # ...
[Service]
CPUAffinity=0,6,7
# ...
      Copy to Clipboard Toggle word wrap

    • Update the CPUAffinity value in the MicroShift ovs-vswitchd systemd file in the path /etc/systemd/system/ovs-vswitchd.service.d/microshift-cpuaffinity.conf.

      MicroShift ovs-vswitchd systemd configuration example

      # ...
[Service]
CPUAffinity=0,6,7
# ...
      Copy to Clipboard Toggle word wrap

    • Update the CPUAffinity value in the MicroShift ovsdb-server systemd file in the path /etc/systemd/system/ovsdb-server.service.d/microshift-cpuaffinity.conf

      MicroShift ovsdb-server systemd configuration example

      # ...
[Service]
CPUAffinity=0,6,7
# ...
      Copy to Clipboard Toggle word wrap

Legal Notice
Copy link

Copyright © 2025 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat