Chapter 7. Configuring MicroShift authentication and security

1. Copy the certificates and keys to the preferred directory in the host operating system. Ensure that the files are accessible only with root access.

2. Update the MicroShift configuration for each custom CA by specifying the certificate names and new fully qualified domain name (FQDN) in the MicroShift /etc/microshift/config.yaml configuration file.

   Each certificate configuration can contain the following values:

   • The certificate file location is a required value.

   • A single common name containing the API server DNS and IP address or IP address range.

     Tip

     In most cases, MicroShift generates a new kubeconfig file for your custom CA that includes the IP address or range that you specify. The exception is when you specify wildcards for the IP address. In this case, MicroShift generates a kubeconfig file with the public IP address of the server. To use wildcards, you must update the kubeconfig file with your specific details.

   • Multiple Subject Alternative Names (SANs) containing the API server DNS and IP addresses or a wildcard certificate.
   • You can list additional DNS names for each certificate.
3. After the MicroShift service restarts, you must copy the generated kubeconfig files to the client.

4. Configure additional CAs on the client system. For example, you can update CA bundles in the Red Hat Enterprise Linux (RHEL) truststore.

   Important

   Custom server certificates must be validated against CA data configured in the trust root of the host operating system. For more information, read the following documentation:

   • The system-wide truststore

5. The certificates and keys are read from the specified file location on the host. You can test and validate configuration from the client.

   • If any validation fails, MicroShift skips the custom configuration and uses the default certificate to start. The priority is to continue the service uninterrupted. MicroShift logs errors when the service starts. Common errors include expired certificates, missing files, or wrong IP addresses.
6. External server certificates are not automatically renewed. You must manually rotate your external certificates.

Procedure

1. Copy the custom certificates you want to add to the trust root of the MicroShift host. Ensure that the certificate and private keys are only accessible to MicroShift.

2. For each custom CA that you need, add an apiServer section called namedCertificates to the /etc/microshift/config.yaml MicroShift configuration file by using the following example:

   apiServer:
     namedCertificates:
      - certPath: ~/certs/api_fqdn_1.crt 

   1

   
        keyPath:  ~/certs/api_fqdn_1.key 

   2

   
      - certPath: ~/certs/api_fqdn_2.crt
        keyPath:  ~/certs/api_fqdn_2.key
        names: 

   3

   
        - api_fqdn_1
        - *.apps.external.com

   Copy to Clipboard Toggle word wrap

   1

   Add the full path to the certificate.

   2

   Add the full path to the certificate key.

   3

   Optional. Add a list of explicit DNS names. Leading wildcards are allowed. If no names are listed, the implicit names are extracted from the certificates.

3. Restart the MicroShift to apply the certificates by running the following command:

   $ systemctl microshift restart

   Copy to Clipboard Toggle word wrap
4. Wait a few minutes for the system to restart and apply the custom server. New kubeconfig files are generated in the /var/lib/microshift/resources/kubeadmin/ directory.
5. Copy the kubeconfig files to the client. If you specified wildcards for the IP address, update the kubeconfig to remove the public IP address of the server and replace that IP address with the specific wildcard range you want to use.

6. From the client, use the following steps:

   1. Specify the kubeconfig to use by running the following command:

      $ export KUBECONFIG=~/custom-kubeconfigs/kubeconfig 

      1

      Copy to Clipboard Toggle word wrap

      1

      Use the location of the copied kubeconfig file as the path.

   2. Check that the certificates are applied by using the following command:

      $ oc --certificate-authority ~/certs/ca.ca get node

      Copy to Clipboard Toggle word wrap

      Example output

      oc get node
      NAME                             STATUS   ROLES                         AGE   VERSION
      dhcp-1-235-195.arm.example.com   Ready    control-plane,master,worker   76m   v1.32.3

      Copy to Clipboard Toggle word wrap

   3. Add the new CA file to the $KUBECONFIG environment variable by running the following command:

      $ oc config set clusters.microshift.certificate-authority /tmp/certificate-authority-data-new.crt

      Copy to Clipboard Toggle word wrap

   4. Verify that the new kubeconfig file contains the new CA by running the following command:

      $ oc config view --flatten

      Copy to Clipboard Toggle word wrap

      Example externally generated kubeconfig file

      apiVersion: v1
      clusters:
      - cluster:
          certificate-authority: /tmp/certificate-authority-data-new.crt 

      1

      
          server: https://api.ci-ln-k0gim2b-76ef8.aws-2.ci.openshift.org:6443
        name: ci-ln-k0gim2b-76ef8
      contexts:
      - context:
          cluster: ci-ln-k0gim2b-76ef8
          user:
        name:
      current-context:
      kind: Config
      preferences: {}

      Copy to Clipboard Toggle word wrap

      1

      The certificate-authority-data section is not present in externally generated kubeconfig files. It is added with the oc config set command used previously.

   5. Verify the subject and issuer of your customized API server certificate authority by running the following command:

      $ curl --cacert /tmp/caCert.pem https://${fqdn_name}:6443/healthz -v

      Copy to Clipboard Toggle word wrap

      Example output

      Server certificate:
        subject: CN=kas-test-cert_server
        start date: Mar 12 11:39:46 2024 GMT
        expire date: Mar 12 11:39:46 2025 GMT
        subjectAltName: host "dhcp-1-235-3.arm.eng.rdu2.redhat.com" matched cert's "dhcp-1-235-3.arm.eng.rdu2.redhat.com"
        issuer: CN=kas-test-cert_ca
        SSL certificate verify ok.

      Copy to Clipboard Toggle word wrap

      Important

      Either replace the certificate-authority-data in the generated kubeconfig file with the new rootCA or add the certificate-authority-data to the trust root of the operating system. Do not use both methods.

   6. Configure additional CAs in the trust root of the operating system. For example, in the RHEL Client truststore on the client system. The system-wide truststore.

      • Updating the certificate bundle with the configuration that contains the CA is recommended.
      • If you do not want to configure your certificate bundles, you can alternately use the oc login localhost:8443 --certificate-authority=/path/to/cert.crt command, but this method is not preferred.

Procedure

1. From MicroShift, ensure that the certificate is served by the kube-apiserver and verify that the certificate path is appended to the --tls-sni-cert-key FLAG by running the following command:

   $ journalctl -u microshift -b0 | grep tls-sni-cert-key

   Copy to Clipboard Toggle word wrap

   Example output

   Jan 24 14:53:00 localhost.localdomain microshift[45313]: kube-apiserver I0124 14:53:00.649099   45313 flags.go:64] FLAG: --tls-sni-cert-key="[/home/eslutsky/dev/certs/server.crt,/home/eslutsky/dev/certs/server.key;/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.key;/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.key;/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.key

   Copy to Clipboard Toggle word wrap

2. From the client, ensure that the kube-apiserver is serving the correct certificate by running the following command:

   $ openssl s_client -connect <SNI_ADDRESS>:6443 -showcerts | openssl x509 -text -noout -in - | grep -C 1 "Alternative\|CN"

   Copy to Clipboard Toggle word wrap

Procedure

1. Stop the MicroShift services and clean up the custom certificates by running the following command:

   $ sudo microshift-cleanup-data --cert

   Copy to Clipboard Toggle word wrap

   Example output

   Stopping MicroShift services
   Removing MicroShift certificates
   MicroShift service was stopped
   Cleanup succeeded

   Copy to Clipboard Toggle word wrap

2. Restart the MicroShift services to recreate the custom certificates by running the following command:

   $ sudo systemctl start microshift

   Copy to Clipboard Toggle word wrap

Procedure

1. Make a copy of the provided config.yaml.default file in the /etc/microshift/ directory, renaming it config.yaml.

2. Keep the new MicroShift config.yaml in the /etc/microshift/ directory. Your config.yaml file is read every time the MicroShift service starts.

   Note

   After you create it, the config.yaml file takes precedence over built-in settings.

3. Optional: Use a configuration snippet if you are using an existing MicroShift YAML. See "Using configuration snippets" in the Additional resources section for more information.

4. Replace the default values in the tls section of the MicroShift YAML with your valid values.

   Example TLS 1.2 configuration

   apiServer:
   # ...
     tls:
       cipherSuites: 

   1

   
       - <cipher_suite_1> 

   2

   
       - ...
       minVersion: VersionTLS12 

   3

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Defaults to the suites of the configured minVersion. If minVersion is not configured, the default value is TLS 1.2.

   2

   Specify the cipher suites you want to use from the list of supported cipher suites. If you do not configure this list, all of the supported cipher suites are used. All clients connecting to the API server must support the configured cipher suites or the connections fail during the TLS handshake phase. Be sure to add the CA certificate bundle to the list of CA certificates that the TLS client or server trusts.

   3

   Specify VersionTLS12 or VersionTLS13.

   Important

   When you choose TLS 1.3 as the minimum TLS version, only the default MicroShift cipher suites can be used. Additional cipher suites are not configurable. If other cipher suites to use with TLS 1.3 are configured, those suites are ignored and overwritten by the MicroShift defaults.

5. Complete any other additional configurations that you require, then restart MicroShift by running the following command:

   $ sudo systemctl restart microshift

   Copy to Clipboard Toggle word wrap

Procedure

1. Make a copy of the provided config.yaml.default file in the /etc/microshift/ directory, renaming it config.yaml. Keep the new MicroShift config.yaml you create in the /etc/microshift/ directory. The new config.yaml is read whenever the MicroShift service starts. After you create it, the config.yaml file takes precedence over built-in settings.

2. Replace the default values in the auditLog section of the YAML with your desired valid values.

   Example default auditLog configuration

   apiServer:
   # ....
     auditLog:
       maxFileAge: 7 

   1

   
       maxFileSize: 200 

   2

   
       maxFiles: 1 

   3

   
       profile: Default 

   4

   
   # ....

   Copy to Clipboard Toggle word wrap

   1

   Specifies the maximum time in days that log files are kept. Files older than this limit are deleted. In this example, after a log file is more than 7 days old, it is deleted. The files are deleted regardless of whether or not the live log has reached the maximum file size specified in the maxFileSize field. File age is determined by the timestamp written in the name of the rotated log file, for example, audit-2024-05-16T17-03-59.994.log. When the value is 0, the limit is disabled.

   2

   The maximum audit log file size in megabytes. In this example, the file is rotated as soon as the live log reaches the 200 MB limit. When the value is set to 0, the limit is disabled.

   3

   The maximum number of rotated audit log files retained. After the limit is reached, the log files are deleted in order from oldest to newest. In this example, the value 1 results in only 1 file of size maxFileSize being retained in addition to the current active log. When the value is set to 0, the limit is disabled.

   4

   Logs only metadata for read and write requests; does not log request bodies except for OAuth access token requests. If you do not specify this field, the Default profile is used.

3. Optional: To specify a new directory for logs, you can stop MicroShift, and then move the /var/log/kube-apiserver directory to your desired location:

   1. Stop MicroShift by running the following command:

      $ sudo systemctl stop microshift

      Copy to Clipboard Toggle word wrap

   2. Move the /var/log/kube-apiserver directory to your desired location by running the following command:

      $ sudo mv /var/log/kube-apiserver <~/kube-apiserver> 

      1

      Copy to Clipboard Toggle word wrap

      1

      Replace <~/kube-apiserver> with the path to the directory that you want to use.

   3. If you specified a new directory for logs, create a symlink to your custom directory at /var/log/kube-apiserver by running the following command:

      $ sudo ln -s <~/kube-apiserver> /var/log/kube-apiserver 

      1

      Copy to Clipboard Toggle word wrap

      1

      Replace <~/kube-apiserver> with the path to the directory that you want to use. This enables the collection of logs in sos reports.

4. If you are configuring audit log policies on a running instance, restart MicroShift by entering the following command:

   $ sudo systemctl restart microshift

   Copy to Clipboard Toggle word wrap

Procedure

1. Download the relevant public key and save it as /etc/containers/RedHat_ReleaseKey3.pub by running the following command:

   $ sudo curl -sL https://access.redhat.com/security/data/63405576.txt -o /etc/containers/RedHat_ReleaseKey3.pub

   Copy to Clipboard Toggle word wrap

2. To configure the container runtime to verify images from Red Hat sources, edit the /etc/containers/policy.json file to contain the following configuration:

   Example policy JSON file

   {
       "default": [
           {
               "type": "reject"
           }
       ],
       "transports": {
           "docker": {
               "quay.io/openshift-release-dev": [{
                   "type": "sigstoreSigned",
                   "keyPath": "/etc/containers/RedHat_ReleaseKey3.pub",
                   "signedIdentity": {
                       "type": "matchRepoDigestOrExact"
                   }
               }],
               "registry.redhat.io": [{
                   "type": "sigstoreSigned",
                   "keyPath": "/etc/containers/RedHat_ReleaseKey3.pub",
                   "signedIdentity": {
                       "type": "matchRepoDigestOrExact"
                   }
               }]
           }
       }
   }

   Copy to Clipboard Toggle word wrap

3. Configure Red Hat remote registries to use sigstore attachments when pulling images to the local storage, by editing the /etc/containers/registries.d/registry.redhat.io.yaml` file to contain the following configuration:

   $ cat /etc/containers/registries.d/registry.redhat.io.yaml
   docker:
        registry.redhat.io:
            use-sigstore-attachments: true

   Copy to Clipboard Toggle word wrap

4. Configure Red Hat remote registries to use sigstore attachments when pulling images to the local storage, by editing the /etc/containers/registries.d/registry.quay.io.yaml file to contain the following configuration:

   $ cat /etc/containers/registries.d/quay.io.yaml
   docker:
     quay.io/openshift-release-dev:
       use-sigstore-attachments: true

   Copy to Clipboard Toggle word wrap
5. Create user-specific registry configuration files if your use case requires signature verification for those image sources. You can use the example here to start with and add your own requirements.

Next steps

1. If you are using a mirror registry, enable sigstore attachments.
2. Otherwise, proceed to wiping the local container storage clean.