Chapter 3. Using ingress control for a MicroShift cluster

Procedure

1. Apply ingress control settings in one of the two following ways:

   1. Update the MicroShift config.yaml configuration file by making a copy of the provided config.yaml.default file in the /etc/microshift/ directory, naming it config.yaml and keeping it in the source directory.

      Important

      After you create the config.yaml, the configuration file takes precedence over built-in settings. The config.yaml file is read every time the MicroShift service starts.

   2. Use a configuration snippet to apply the ingress control settings you want. To do this, create a configuration snippet YAML file and put it in the /etc/microshift/config.d/ configuration directory.

      Important

      Configuration snippet YAMLs take precedence over both built-in settings and the config.yaml configuration file.

2. Replace the default values in the ingress section of the MicroShift YAML with your valid values, or create a configuration snippet file with the sections you need.

   Ingress controller configuration fields with default values

   apiServer:
   # ...
   ingress:
     certificateSecret: router-certs-custom
     clientTLS:
       allowedSubjectPatterns: []
       clientCA:
         name: ""
       clientCertificatePolicy: ""
     defaultHTTPVersion: 1
     forwardedHeaderPolicy: Append
     httpCompression:
       mimeTypes:
         - ""
     httpEmptyRequestsPolicy: Respond
     listenAddress: []
     logEmptyRequests: Log
     ports:
        http: 80
        https: 443
     routeAdmissionPolicy:
       namespaceOwnership: InterNamespaceAllowed
       wildcardPolicy: WildcardsDisallowed
     status: Managed
     tlsSecurityProfile:
       type:
       custom:
         ciphers:[]
         minTLSVersion:""
       intermediate: {}
       old: {}
       type: ""
     tuningOptions:
       clientFinTimeout: 1s
       clientTimeout: 30s
       headerBufferBytes: 0
       headerBufferMaxRewriteBytes: 0
       healthCheckInterval: 5s
       maxConnections: 0
       serverFinTimeout: 1s
       serverTimeout: 30s
       threadCount: 4
       tlsInspectDelay: 5s
       tunnelTimeout: 1h
   # ...

   Copy to Clipboard

   Table 3.1. Ingress controller configuration fields definitions table

   Parameter	Description
   ingress	The ingress section of the MicroShift config.yaml file defines the configurable parameters for the implemented portions of the OpenShift Container Platform Ingress Control Operator. All parameters in the rest of this table are subsections in the ingress section of the config.yaml.
   certificateSecret	A reference to a kubernetes.io/tls type of secret that contains the default certificate that is served by the MicroShift ingress controller. When routes do not specify their own certificate, the certificateSecret parameter is used. All secrets used must contain tls.key key file contents and tls.crt certificate file contents. When the certificateSecret parameter is not set, a wildcard certificate is automatically generated and used. The wildcard certificate is valid for the ingress controller default domain and its subdomains. The generated certificate authority (CA) is automatically integrated with the truststore of the cluster. In-use generated and user-specified certificates are automatically integrated with the MicroShift built-in OAuth server.
   clientTLS	Authenticates client access to the cluster and services. As a result, mutual TLS authentication is enabled. If not set, then client TLS is not enabled. You must set the spec.clientTLS.clientCertificatePolicy and spec.clientTLS.ClientCA parameters to use client TLS.
   clientTLS.AllowedSubjectPatterns	Optional subfield that specifies a list of regular expressions that are matched against the distinguished name on a valid client certificate to filter requests. Use this parameter to cause the ingress controller to reject certificates based on the distinguished name. The Perl Compatible Regular Expressions (PCRE) syntax is required. Important When configured, this field must contain a valid expression or the MicroShift service fails. At least one pattern must match a client certificate’s distinguished name; otherwise, the ingress controller rejects the certificate and denies the connection.
   clientTLS.ClientCA	Specifies a required config map that is in the openshift-ingress namespace. Required to enable client TLS. The config map must contain a certificate authority (CA) bundle.
   clientTLS.ClientCA.name	The metadata.name of the config map referenced in the clientTLS.ClientCA value.
   clientTLS.ClientCertificatePolicy	Required or Optional are valid values. Set to Required to enable client TLS. The ingress controller only checks client certificates for edge-terminated and re-encrypted TLS routes. The ingress controller cannot check certificates for plain text HTTP or passthrough TLS routes.
   defaultHTTPVersion	Sets the HTTP version for the ingress controller. Default value is 1 for HTTP 1.1.
   forwardedHeaderPolicy	Specifies when and how the ingress controller sets the Forwarded, X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Port, X-Forwarded-Proto, and X-Forwarded-Proto-Version HTTP headers. The following values are valid: Append preserves any existing headers by specifying that the ingress controller appends them. 'Append` is the default value. Replace removes any existing headers by specifying that the ingress controller sets the headers. IfNone sets the headers set by specifying that the ingress controller sets the headers if they are not already set. Never preserves any existing headers by specifying that the ingress controller never sets the headers.
   httpCompression	Defines the policy for HTTP traffic compression.
   httpCompression.mimeTypes	Defines a list of MIME types to which compression should be applied. For example, text/css; charset=utf-8, text/html, text/*, image/svg+xml, application/octet-stream, X-custom/customsub, in the, type/subtype; [;attribute=value] format. Valid types are: application, image, message, multipart, text, video, or a custom type prefaced by X-. To see the full notation for MIME types and subtypes, see RFC1341 (IETF Datatracker documentation).
   httpEmptyRequestsPolicy	Describes how HTTP connections are handled if the connection times out before a request is received. Allowed values for this field are Respond and Ignore. The default value is Respond. Empty requests typically come from load-balancer health probes or preconnects and can often be safely ignored. However, these requests can also be caused by network errors and port scans. Therefore, setting this field to Ignore can impede detection or diagnosis of network problems and detecting intrusion attempts. When the policy is set to Respond, the ingress controller sends an HTTP 400 or 408 response, logs the connection if access logging is enabled, and counts the connection in the appropriate metrics. When the policy is set to Ignore, the http-ignore-probes parameter is added to the HAproxy process configuration. After this parameter is added, the ingress controller closes the connection without sending a response, then either logs the connection or incrementing metrics.
   logEmptyRequests	Specifies connections for which no request is received and logged. Log and Ignore are valid values. Empty requests typically come from load-balancer health probes or preconnects and can often be safely ignored. However, these requests can also be caused by network errors and port scans. Therefore, setting this field to Ignore can impede detection or diagnosis of network problems and detecting intrusion attempts. The default value is Log. Setting this value to Log indicates that an event should be logged. Setting this value to Ignore sets the dontlognull option in the HAproxy configuration.
   ports	Defines default router ports.
   ports.http	Default router http port. Must be in range 1-65535. Default value is 80.
   ports.https	Default router https port. Must be in range 1-65535. Default value is 443.
   routeAdmission	Defines a policy for handling new route claims, such as allowing or denying claims across namespaces.
   routeAdmission.namespaceOwnership	Describes how hostname claims across namespaces should be handled. The default is InterNamespaceAllowed. The following are valid values: Strict does not allow routes to claim the same hostname across namespaces. InterNamespaceAllowed allows routes to claim different paths of the same hostname across namespaces.
   routeAdmission.wildcardPolicy	Controls how routes with configured wildcard policies are handled by the ingress controller. WildcardsAllowed and WildcardsDisallowed are valid values. Default value is WildcardsDisallowed. WildcardPolicyAllowed means that routes with any wildcard policy are admitted by the ingress controller. WildcardPolicyDisallowed means that only routes with a wildcard policy of None are admitted by the ingress controller. Important Changing the wildcard policy from WildcardsAllowed to WildcardsDisallowed causes admitted routes with a wildcard policy of subdomain to stop working. These routes must be recreated to a wildcard policy of None to be readmitted by the ingress controller.
   status	Default router status. Managed or Removed are valid values.
   tlsSecurityProfile	tlsSecurityProfile specifies settings for TLS connections for ingress controllers. If not set, the default value is based on the apiservers.config.openshift.io/cluster resource. The TLS 1.0 version of an Old or Custom profile is automatically converted to 1.1 by the ingress controller. Intermediate is the default setting. The minimum TLS version for ingress controllers is 1.1. The maximum TLS version is 1.3. Note Ciphers and the minimum TLS version of the configured security profile are reflected in the TLSProfile status. Profiles are intent-based and change over time when new ciphers are developed and existing ciphers are found to be insecure. The usable list can be reduced depending on which ciphers are available to a specific process.
   tlsSecurityProfile.custom	User-defined TLS security profile. If you configure this parameter and related parameters, use extreme caution.
   tlsSecurityProfile.custom.ciphers	Specifies the cipher algorithms that are negotiated during the TLS handshake. Operators might remove entries their operands do not support.
   tlsSecurityProfile.custom.minTLSVersion	Specifies the minimal version of the TLS protocol that is negotiated during the TLS handshake. For example, to use TLS versions 1.1, 1.2 and 1.3, set the value to VersionTLS11. The highest valid value for minTLSVersion is VersionTLS12.
   tlsSecurityProfile.intermediate	This TLS profile can be used for a majority of services. Intermediate compatibility (recommended).
   tlsSecurityProfile.old	Used for backward compatibility. Old backward compatibility.
   tlsSecurityProfile.type	Valid values are Intermediate, Old, or Custom. The Modern value is not supported.
   tuningOptions	Specifies options for tuning the performance of ingress controller pods.
   tuningOptions.clientFinTimeout	Specifies how long a connection is held open while waiting for the client response to the server closing the connection. The default timeout is 1s.
   tuningOptions.clientTimeout	Specifies how long a connection is held open while waiting for a client response. The default timeout is 30s.
   tuningOptions.headerBufferBytes	Specifies how much memory is reserved, in bytes, for ingress controller connection sessions. This value must be at least 16384 if HTTP/2 is enabled for the ingress controller. If not set, the default value is 32768 bytes. Important Setting this field not recommended because headerBufferMaxRewriteBytes parameter values that are too small can break the ingress controller. Conversely, values for headerBufferMaxRewriteBytes that are too large could cause the ingress controller to use significantly more memory than necessary.
   tuningOptions.headerBufferMaxRewriteBytes	Specifies how much memory should be reserved, in bytes, from headerBufferBytes for HTTP header rewriting and appending for ingress controller connection sessions. The minimum value for headerBufferMaxRewriteBytes is 4096. headerBufferBytes must be greater than the headerBufferMaxRewriteBytes value for incoming HTTP requests. If not set, the default value is 8192 bytes. Important Setting this field is not recommended because headerBufferMaxRewriteBytes values that are too small can break the ingress controller and headerBufferMaxRewriteBytes that are too large could cause the ingress controller to use significantly more memory than necessary.
   tuningOptions.healthCheckInterval	Specifies how long the router waits between health checks, set in seconds. The default is 5s.
   tuningOptions.maxConnections	Specifies the maximum number of simultaneous connections that can be established for each HAProxy process. Increasing this value allows each ingress controller pod to handle more connections at the cost of additional system resources. Permitted values are 0, -1, any value within the range 2000 and 2000000, or the field can be left empty. If this field is left empty or has the value 0, the ingress controller uses the default value of 50000. If the field has the value of -1, then the HAProxy process dynamically computes a maximum value based on the available ulimits in the running container. This process results in a large computed value that incurs significant memory usage compared to the current default value of 50000. If the field has a value that is greater than the current operating system limit, the HAProxy processes do not start. If you choose a discrete value and the router pod is migrated to a new node, it is possible that the new node does not have an identical ulimit configured. In such cases, the pod fails to start. If you have nodes with different ulimits configured, and you choose a discrete value, you can use the value of -1 for this field so that the maximum number of connections is calculated at runtime. You can monitor memory usage for router containers with the container_memory_working_set_bytes{container="router",namespace="openshift-ingress"} metric. You can monitor memory usage of individual HAProxy processes in router containers with the container_memory_working_set_bytes{container="router",namespace="openshift-ingress"}/container_processes{container="router",namespace="openshift-ingress"} metric.
   tuningOptions.serverFinTimeout	Specifies how long a connection is held open while waiting for the server response to the client that is closing the connection. The default timeout is 1s.
   tuningOptions.serverTimeout	Specifies how long a connection is held open while waiting for a server response. The default timeout is 30s.
   tuningOptions.threadCount	Specifies the number of threads to create per HAProxy process. Creating more threads allows each ingress controller pod to handle more connections, at the cost of more system resources being used. The HAProxy load balancer supports up to 64 threads. If this field is empty, the ingress controller uses the default value of 4 threads. Important Setting this field is not recommended because increasing the number of HAProxy threads allows ingress controller pods to use more CPU time under load, and prevent other pods from receiving the CPU resources they need to perform. Reducing the number of threads can cause the ingress controller to perform poorly.
   tuningOptions.tlsInspectDelay	Specifies how long the router can hold data to find a matching route. Setting this value too low can cause the router to fall back to the default certificate for edge-terminated, re-encrypted, or passthrough routes, even when using a better-matched certificate. The default inspect delay is 5s.
   tuningOptions.tunnelTimeout	Specifies how long a tunnel connection, including websockets, remains open while the tunnel is idle. The default timeout is 1h.

3. Complete any other configurations you require, then start or restart MicroShift by running one the following commands:

   $ sudo systemctl start microshift

   Copy to Clipboard

   $ sudo systemctl restart microshift

   Copy to Clipboard