Chapter 17. Recovering from expired system certificates

Procedure

1. Verify that the IdM services are running:

   # ipactl status

   Copy to Clipboard Toggle word wrap

2. Optional. If any of the services are not running, force them to start:

   # ipactl start -f

   Copy to Clipboard Toggle word wrap

   The -f or --force flag bypasses some of the startup checks, which is necessary when services cannot communicate due to expired certificates.

3. Compare RA Agent and Subsystem certificates between the renewal server and the failing replica.

   On both the healthy renewal server and the failing replica, retrieve the serial numbers of the RA agent and subsystem certificates, and verify that they are identical.

   1. RA Agent certificate and its corresponding LDAP entry. The serial number is the second value in the semicolon-separated description attribute string, for example 2;SERIAL;…​:

      # openssl x509 -in /var/lib/ipa/ra-agent.pem -noout -serial
      
      # ldapsearch -D "cn=directory manager" -W -b "uid=ipara,ou=people,o=ipaca" description

      Copy to Clipboard Toggle word wrap

   2. Subsystem certificate and its corresponding LDAP entry. The serial number is the second value in the semicolon-separated description attribute string, for example 2;SERIAL;…​:

      # certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'subsystemCert cert-pki-ca' | grep "Serial Number"
      
      # ldapsearch -D "cn=directory manager" -W -b "uid=pkidbuser,ou=People,o=ipaca" description

      Copy to Clipboard Toggle word wrap

      If the serial numbers on the replica do not match those on the CA renewal server, proceed with the next steps to synchronize them.

4. Manually copy the healthy certificates to the replica. From the renewal server, copy the ra-agent.pem and the subsystem certificate to the failing replica.

   1. Copy the RA agent certificate file:

      # scp /var/lib/ipa/ra-agent.pem root@failed-replica.idm.example.com:/tmp/

      Copy to Clipboard Toggle word wrap

   2. Export the subsystem certificate:

      # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a > /tmp/subsystem.pem

      Copy to Clipboard Toggle word wrap

   3. Copy the subsystem certificate:

      # scp /tmp/subsystem.pem root@failed-replica.idm.example.com:/tmp/

      Copy to Clipboard Toggle word wrap

5. On the failing replica, replace the expired RA agent certificate.

   1. Copy the healthy certificate into place:

      # cp /tmp/ra-agent.pem /var/lib/ipa/ra-agent.pem

      Copy to Clipboard Toggle word wrap

   2. Update the corresponding entry in LDAP to ensure it contains the correct certificate blob and serial number. This step requires the Directory Manager password.

      1. Prepare the new RA agent certificate for LDAP by converting it to a single-line blob and storing it in a variable:

         # RA_CERT_BLOB=$(sed -rn '/^-----BEGIN CERTIFICATE-----$/{:1;n;/^-----END CERTIFICATE-----$/b2;H;b1};:2;${x;s/\s//g;p}' /tmp/ra-agent.pem)

         Copy to Clipboard Toggle word wrap

      2. Extract the serial number from the new certificate and store it in a variable:

         # RA_CERT_SERIAL=$(openssl x509 -in /tmp/ra-agent.pem -noout -serial | cut -d'=' -f2)

         Copy to Clipboard Toggle word wrap

      3. Ensure the IdM services are running before attempting to modify the LDAP database:

         # ipactl start -f

         Copy to Clipboard Toggle word wrap

      4. Update the LDAP entry for the RA agent with the new certificate blob and serial number. You will be prompted for the Directory Manager password:

         # ldapmodify -D "cn=Directory Manager" -W -x <<EOF
         dn: uid=ipara,ou=people,o=ipaca
         changetype: modify
         add: userCertificate
         userCertificate:: ${RA_CERT_BLOB}
         
         replace: description
         description: 2;${RA_CERT_SERIAL};CN=Certificate Authority,O=IDM.EXAMPLE.COM;CN=IPA RA,O=IDM.EXAMPLE.COM
         EOF

         Copy to Clipboard Toggle word wrap

         Replace IDM.EXAMPLE.COM with your IdM realm or use your custom certificate subject base.

6. On the failing replica, replace the expired subsystem certificate.

   1. Import the healthy certificate into the NSS database. This requires the NSS database password, which can be found in /etc/pki/pki-tomcat/password.conf.

      1. Set a variable for the location of the NSS database password file for easier use in the following commands:

         # PWDFILE=/etc/pki/pki-tomcat/alias/pwdfile.txt

         Copy to Clipboard Toggle word wrap

      2. Remove the old, expired subsystem certificate from the NSS database:

         # certutil -D -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -f $PWDFILE

         Copy to Clipboard Toggle word wrap

      3. Add the new, healthy subsystem certificate from the /tmp/subsystem.pem file to the NSS database:

         # certutil -A -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -t ",," -i /tmp/subsystem.pem -f $PWDFILE

         Copy to Clipboard Toggle word wrap

   2. Update the corresponding entry in LDAP, similar to the RA agent step:

      # SUBSYS_CERT_BLOB=$(cat /tmp/subsystem.pem | sed -rn '/^-----BEGIN CERTIFICATE-----$/{:1;n;/^-----END CERTIFICATE-----$/b2;H;b1};:2;${x;s/\s//g;p}')
      
      # SUBSYS_CERT_SERIAL=$(openssl x509 -in /tmp/subsystem.pem -noout -serial | cut -d'=' -f2)
      
      # ldapmodify -D "cn=Directory Manager" -W -x <<EOF
      dn: uid=pkidbuser,ou=people,o=ipaca
      changetype: modify
      add: userCertificate
      userCertificate:: ${SUBSYS_CERT_BLOB}
      
      replace: description
      description: 2;${SUBSYS_CERT_SERIAL};CN=Certificate Authority,O=IDM.EXAMPLE.COM;CN=CA Subsystem,O=IDM.EXAMPLE.COM
      EOF

      Copy to Clipboard Toggle word wrap

      Replace IDM.EXAMPLE.COM with your IdM realm or use your custom certificate subject base.

7. Manually issue a temporary certificate for the replica’s LDAP service.

   1. On the failing replica, generate a Certificate Signing Request (CSR) from the existing key in the Directory Server’s NSS database:

      # DS_INSTANCE=$(grep ldap_uri /etc/ipa/default.conf | sed -n 's|.%2Frun%2F\(slapd-\)\.socket.|\1|p')*
      
      # LDAP_CERT_NICKNAME=$(grep nsSSLPersonalitySSL "/etc/dirsrv/${DS_INSTANCE}/dse.ldif" | awk '{ print $2 }')
      
      # HOSTNAME=$(hostname -f)
      
      # PWDFILE="/etc/dirsrv/${DS_INSTANCE}/pwdfile.txt"
      
      # certutil -R -d "/etc/dirsrv/${DS_INSTANCE}" -k ${LDAP_CERT_NICKNAME} -n ${LDAP_CERT_NICKNAME} -s "CN=${HOSTNAME}" -a -f "${PWDFILE}" -8 "${HOSTNAME}" -o /tmp/ldap.csr

      Copy to Clipboard Toggle word wrap
      Note

      The command above creates /tmp/ldap.csr. Some versions of certutil may add extra text to the file. Ensure the file contains only the -----BEGIN NEW CERTIFICATE REQUEST----- block and nothing else.

   2. If the certificate is signed by the IdM CA:

      1. On the failing replica, copy the CSR to the renewal server:

         # scp /tmp/ldap.csr root@renewal-master.idm.example.com:/tmp/

         Copy to Clipboard Toggle word wrap

      2. On the renewal server, sign the CSR. Use the ldap/ principal to ensure the certificate is valid for GSSAPI binds:

         # ipa cert-request /tmp/ldap.csr --principal="ldap/$(hostname -f)" --certificate-out=/tmp/ldap.pem

         Copy to Clipboard Toggle word wrap

      3. On the renewal server, copy the new certificate back to the replica:

         # scp /tmp/ldap.pem root@failed-replica.idm.example.com:/tmp/

         Copy to Clipboard Toggle word wrap

      4. On the failing replica, import the new certificate into the Directory Server’s NSS database:

         # certutil -D -d "/etc/dirsrv/${DS_INSTANCE}" -n ${LDAP_CERT_NICKNAME} -f "${PWDFILE}"
         
         # certutil -A -d "/etc/dirsrv/${DS_INSTANCE}" -n ${LDAP_CERT_NICKNAME} -t ",," -i /tmp/ldap.pem -f "${PWDFILE}"

         Copy to Clipboard Toggle word wrap

   3. If the certificate is signed by an external CA:

      1. Take the /tmp/ldap.csr file and submit it to your external Certificate Authority for signing.
      2. Once you receive the new certificate file (for example, new_ldap_cert.pem), copy it securely to the /tmp/ directory on the failing replica and rename it to ldap.pem.

      3. On the failing replica, import the new certificate into the Directory Server’s NSS database:

         # certutil -D -d "/etc/dirsrv/${DS_INSTANCE}" -n ${ LDAP_CERT_NICKNAME} -f "${PWDFILE}"
         
         # certutil -A -d "/etc/dirsrv/${DS_INSTANCE}" -n ${ LDAP_CERT_NICKNAME} -t ",," -i /tmp/ldap.pem -f "${PWDFILE}"

         Copy to Clipboard Toggle word wrap

8. Manually issue a temporary certificate for the replica’s PKI Tomcat service.

   1. On the affected replica, generate a CSR using the existing key stored in the PKI Tomcat NSS database:

      # HOSTNAME=$(hostname -f)
      
      # PWDFILE="/etc/pki/pki-tomcat/alias/pwdfile.txt"
      
      # certutil -R -d /etc/pki/pki-tomcat/alias -k 'Server-Cert cert-pki-ca' -n 'Server-Cert cert-pki-ca' -s "CN=${HOSTNAME},O=IDM.EXAMPLE.COM" -a -f "${PWDFILE}" -o /tmp/server-cert.csr

      Copy to Clipboard Toggle word wrap

      Replace IDM.EXAMPLE.COM with your IdM realm or use your custom certificate subject base.

   2. On the failing replica, copy the CSR to the renewal server:

      # scp /tmp/server-cert.csr root@renewal-master.idm.example.com:/tmp/

      Copy to Clipboard Toggle word wrap

   3. On the renewal server, sign the CSR. Use the host/ principal for the web server certificate:

      # ipa cert-request /tmp/server-cert.csr --principal="host/$(hostname -f)" --certificate-out=/tmp/server-cert.pem

      Copy to Clipboard Toggle word wrap

   4. On the renewal server, copy the new certificate back to the replica:

      # scp /tmp/server-cert.pem root@failed-replica.idm.example.com:/tmp/

      Copy to Clipboard Toggle word wrap

   5. On the failing replica, import the new certificate into the PKI Tomcat NSS database:

      # certutil -D -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' -f "${PWDFILE}"
      
      # certutil -A -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' -t ",," -i /tmp/server-cert.pem -f "${PWDFILE}"

      Copy to Clipboard Toggle word wrap

9. Restart services and renew remaining certificates.

   1. Restart the IdM services on the replica to use the new certificates:

      # ipactl restart

      Copy to Clipboard Toggle word wrap

   2. Restart the certmonger service to apply the manual changes. Once restarted, it should begin attempting to renew any remaining certificates:

      # systemctl restart certmonger

      Copy to Clipboard Toggle word wrap
   3. You can watch the renewal process by periodically running getcert list | grep -E "Request ID|status|expires". Once all certificates are in the MONITORING state, the process is complete.

   4. The certmonger service should now be able to communicate with the CA. If one or more certificates remain unrenewed after a few minutes, consider renewing them manually:

      # getcert list
      
      # getcert resubmit -i <REQUEST_ID>

      Copy to Clipboard Toggle word wrap

Procedure

1. Stop IdM services on the renewal server:

   # ipactl stop

   Copy to Clipboard Toggle word wrap

2. Run ipa-cert-fix to:

   • Inspect tracked certificates.
   • Resets the certificates' expiration dates in LDAP.

   • Allow certmonger to issue new certificates.

     Type yes to continue when prompted:

     # ipa-cert-fix

     Copy to Clipboard Toggle word wrap
     Warning

     Do not run the ipa-cert-fix command if your CA is expired. In that case, skip this step.

3. Run ipa-certupdate to propagate the newly fixed and issued certificates to all system services:

   # ipa-certupdate

   Copy to Clipboard Toggle word wrap

4. Restart IdM and verify:

   # ipactl restart

   Copy to Clipboard Toggle word wrap

Procedure

1. On the renewal server, renew the CA certificate as self-signed. This command uses the existing private key to generate a new CA certificate with a renewed validity period:

   # ipa-cacert-manage renew --self-signed

   Copy to Clipboard Toggle word wrap

   If your certificate is close to its expiry date and renewing it did not push the expiry date forward, see the When IPA’s CA certificate is close to expiry date, renewing it doesn’t push forward the expiry date solution.

2. Fix expired service certificates on the renewal server.

   1. Stop the IdM services:

      # ipactl stop

      Copy to Clipboard Toggle word wrap

   2. Run the ipa-cert-fix command to reset the expiration date of expired service certificates in the IdM database:

      # ipa-cert-fix

      Copy to Clipboard Toggle word wrap

   3. Issue new certificates and apply them to the IdM services:

      # ipa-certupdate

      Copy to Clipboard Toggle word wrap

   4. Restart the IdM services to complete the renewal:

      # ipactl start

      Copy to Clipboard Toggle word wrap

3. If your original setup used an externally-signed CA, you must now get the new CA certificate re-signed by your external authority. If you use a self-signed CA, skip this step.

   1. Generate a certificate-signing request (CSR) for the newly renewed CA:

      # ipa-cacert-manage renew --external-ca

      Copy to Clipboard Toggle word wrap

      This creates a CSR file at /var/lib/ipa/ca.csr.

   2. Submit the CSR file to your external CA and receive the newly signed certificate and the external CA’s chain of trust.

   3. Install the newly signed certificate back into IdM:

      # ipa-cacert-manage renew --external-cert-file /path/to/new-ca.pem --external-cert-file /path/to/external-chain.pem

      Copy to Clipboard Toggle word wrap

4. Propagate the new CA certificate to all IdM servers and clients. Run the ipa-certupdate on the renewal server first, then on all other replicas, and finally on all clients:

   # ipa-certupdate

   Copy to Clipboard Toggle word wrap

   This step is critical to ensure all systems in the topology trust the new CA certificate.