Chapter 3. Configuring custom certificate authorities
You can encrypt connections by using custom certificate authorities (CAs) with the MicroShift service.
3.1. How custom certificate authorities work in MicroShift
The default API server certificate is issued by an internal MicroShift cluster certificate authority (CA). Clients outside of the cluster cannot verify the API server certificate by default. This certificate can be replaced by a custom server certificate that is issued externally by a custom CA that clients trust. The following steps illustrate the workflow in MicroShift:
- Copy the certificates and keys to the preferred directory in the host operating system. Ensure that the files are accessible by root only.
Update the MicroShift configuration for each custom CA by specifying the certificate names and new fully qualified domain name (FQDN) in the MicroShift
/etc/microshift/config.yaml
configuration file.Each certificate configuration can contain the following values:
- The certificate file location is a required value.
A single common name containing the API server DNS and IP address or IP address range.
TipIn most cases, MicroShift generates a new
kubeconfig
for your custom CA that includes the IP address or range that you specify. The exception is when wildcards are specified for the IP address. In this case, MicroShift generates akubeconfig
with the public IP address of the server. To use wildcards, you must update thekubeconfig
file with your specific details.- Multiple Subject Alternative Names (SANs) containing the API server DNS and IP addresses or a wildcard certificate.
- You can provide additional DNS names for each certificate.
-
After the MicroShift service restarts, you must copy the generated
kubeconfig
files to the client. - Configure additional CAs on the client system. For example, you can update CA bundles in the Red Hat Enterprise Linux (RHEL) truststore.
- The certificates and keys are read from the specified file location on the host. Testing and validation of configuration is done from the client.
- External server certificates are not automatically renewed. You must manually rotate your external certificates.
If any validation fails, the MicroShift service skips the custom configuration and uses the default certificate to start. The priority is to continue the service uninterrupted. MicroShift logs errors when the service starts. Common errors include expired certificates, missing files, or incorrect IP addresses.
Custom server certificates have to be validated against CA data configured in the trust root of the host operating system. For information, see The system-wide truststore.
3.2. Configuring custom certificate authorities
To configure externally generated certificates and domain names using custom certificate authorities (CAs), add them to the MicroShift /etc/microshift/config.yaml
configuration file. You must also configure the host operating system trust root.
Externally generated kubeconfig
files are created in the /var/lib/microshift/resources/kubeadmin/<hostname>/kubeconfig
directory. If you need to use localhost
in addition to externally generated configurations, retain the original kubeconfig
file in its default location. The localhost
kubeconfig
file uses the self-signed certificate authority.
Prerequisites
-
The OpenShift CLI (
oc
) is installed. - You have access to the cluster as a user with the cluster administration role.
- The certificate authority has issued the custom certificates.
-
A MicroShift
/etc/microshift/config.yaml
configuration file exists.
Procedure
- Copy the custom certificates you want to add to the trust root of the MicroShift host. Ensure that the certificate and private keys are only accessible to MicroShift.
For each custom CA that you need, add an
apiServer
section callednamedCertificates
to the/etc/microshift/config.yaml
MicroShift configuration file by using the following example:apiServer: namedCertificates: - certPath: ~/certs/api_fqdn_1.crt 1 keyPath: ~/certs/api_fqdn_1.key 2 - certPath: ~/certs/api_fqdn_2.crt keyPath: ~/certs/api_fqdn_2.key names: 3 - api_fqdn_1 - *.apps.external.com
Restart the {microshift-service} to apply the certificates by running the following command:
$ systemctl microshift restart
-
Wait a few minutes for the system to restart and apply the custom server. New
kubeconfig
files are generated in the/var/lib/microshift/resources/kubeadmin/
directory. -
Copy the
kubeconfig
files to the client. If you specified wildcards for the IP address, update thekubeconfig
to remove the public IP address of the server and replace that IP address with the specific wildcard range you want to use. From the client, use the following steps:
Specify the
kubeconfig
to use by running the following command:$ export KUBECONFIG=~/custom-kubeconfigs/kubeconfig 1
- 1
- Use the location of the copied
kubeconfig
file as the path.
Check that the certificates are applied by using the following command:
$ oc --certificate-authority ~/certs/ca.ca get node
Example output
oc get node NAME STATUS ROLES AGE VERSION dhcp-1-235-195.arm.example.com Ready control-plane,master,worker 76m v1.29.2
Add the new CA file to the $KUBECONFIG environment variable by running the following command:
$ oc config set clusters.microshift.certificate-authority /tmp/certificate-authority-data-new.crt
Verify that the new
kubeconfig
file contains the new CA by running the following command:$ oc config view --flatten
Example externally generated
kubeconfig
fileapiVersion: v1 clusters: - cluster: certificate-authority: /tmp/certificate-authority-data-new.crt 1 server: https://api.ci-ln-k0gim2b-76ef8.aws-2.ci.openshift.org:6443 name: ci-ln-k0gim2b-76ef8 contexts: - context: cluster: ci-ln-k0gim2b-76ef8 user: name: current-context: kind: Config preferences: {}
- 1
- The
certificate-authority-data
section is not present in externally generatedkubeconfig
files. It is added with theoc config set
command used previously.
Verify the
subject
andissuer
of your customized API server certificate authority by running the following command:$ curl --cacert /tmp/caCert.pem https://${fqdn_name}:6443/healthz -v
Example output
Server certificate: subject: CN=kas-test-cert_server start date: Mar 12 11:39:46 2024 GMT expire date: Mar 12 11:39:46 2025 GMT subjectAltName: host "dhcp-1-235-3.arm.eng.rdu2.redhat.com" matched cert's "dhcp-1-235-3.arm.eng.rdu2.redhat.com" issuer: CN=kas-test-cert_ca SSL certificate verify ok.
ImportantEither replace the
certificate-authority-data
in the generatedkubeconfig
file with the newrootCA
or add thecertificate-authority-data
to the trust root of the operating system. Do not use both methods.Configure additional CAs in the trust root of the operating system. For example, in the RHEL Client truststore on the client system. See The system-wide truststore for details.
- Updating the certificate bundle with the configuration that contains the CA is recommended.
-
If you do not want to configure your certificate bundles, you can alternately use the
oc login localhost:8443 --certificate-authority=/path/to/cert.crt
command, but this method is not preferred.
3.3. Custom certificates reserved name values
The following certificate problems cause MicroShift to ignore certificates dynamically and log an error:
- The certificate files do not exist on the disk or are not readable.
- The certificate is not parsable.
-
The certificate overrides the internal certificates IPAddress/DNSNames in a
SubjectAlternativeNames
(SAN) field. Do not use a reserved name when configuring SANs.
Address | Type | Comment |
---|---|---|
| DNS | |
| IP Address | |
| IP Address | Cluster Network |
| IP Address | Service Network |
169.254.169.2/29 | IP Address | br-ex Network |
| DNS | |
| DNS | |
| DNS |
3.4. Troubleshooting custom certificates
To troubleshoot the implementation of custom certificates, you can take the following steps.
Procedure
From MicroShift, ensure that the certificate is served by the
kube-apiserver
and verify that the certificate path is appended to the--tls-sni-cert-key
FLAG by running the following command:$ journalctl -u microshift -b0 | grep tls-sni-cert-key
Example output
Jan 24 14:53:00 localhost.localdomain microshift[45313]: kube-apiserver I0124 14:53:00.649099 45313 flags.go:64] FLAG: --tls-sni-cert-key="[/home/eslutsky/dev/certs/server.crt,/home/eslutsky/dev/certs/server.key;/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.key;/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.key;/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.key
From the client, ensure that the
kube-apiserver
is serving the correct certificate by running the following command:$ openssl s_client -connect <SNI_ADDRESS>:6443 -showcerts | openssl x509 -text -noout -in - | grep -C 1 "Alternative\|CN"
3.5. Cleaning up and recreating the custom certificates
To stop the MicroShift services, clean up the custom certificates and recreate the custom certificates, use the following steps.
Procedure
Stop the MicroShift services and clean up the custom certificates by running the following command:
$ sudo microshift-cleanup-data --cert
Example output
Stopping MicroShift services Removing MicroShift certificates MicroShift service was stopped Cleanup succeeded
Restart the MicroShift services to recreate the custom certificates by running the following command:
$ sudo systemctl start microshift