22장. Recovering from expired system certificates

Procedure

1. Verify that the IdM services are running:

   # ipactl status

2. Optional. If any of the services are not running, force them to start:

   # ipactl start -f

   The -f or --force flag bypasses some of the startup checks, which is necessary when services cannot communicate due to expired certificates.

3. Compare RA Agent and Subsystem certificates between the renewal server and the failing replica.

   On both the healthy renewal server and the failing replica, retrieve the serial numbers of the RA agent and subsystem certificates, and verify that they are identical.

   1. RA Agent certificate and its corresponding LDAP entry. The serial number is the second value in the semicolon-separated description attribute string, for example 2;SERIAL;…​:

      # openssl x509 -in /var/lib/ipa/ra-agent.pem -noout -serial
      
      # ldapsearch -D "cn=directory manager" -W -b "uid=ipara,ou=people,o=ipaca" description

   2. Subsystem certificate and its corresponding LDAP entry. The serial number is the second value in the semicolon-separated description attribute string, for example 2;SERIAL;…​:

      # certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'subsystemCert cert-pki-ca' | grep "Serial Number"
      
      # ldapsearch -D "cn=directory manager" -W -b "uid=pkidbuser,ou=People,o=ipaca" description

      If the serial numbers on the replica do not match those on the CA renewal server, proceed with the next steps to synchronize them.

4. Manually copy the healthy certificates to the replica. From the renewal server, copy the ra-agent.pem and the subsystem certificate to the failing replica.

   1. Copy the RA agent certificate file:

      # scp /var/lib/ipa/ra-agent.pem root@failed-replica.idm.example.com:/tmp/

   2. Export the subsystem certificate:

      # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a > /tmp/subsystem.pem

   3. Copy the subsystem certificate:

      # scp /tmp/subsystem.pem root@failed-replica.idm.example.com:/tmp/

5. On the failing replica, replace the expired RA agent certificate.

   1. Copy the healthy certificate into place:

      # cp /tmp/ra-agent.pem /var/lib/ipa/ra-agent.pem

   2. Update the corresponding entry in LDAP to ensure it contains the correct certificate blob and serial number. This step requires the Directory Manager password.

      1. Prepare the new RA agent certificate for LDAP by converting it to a single-line blob and storing it in a variable:

         # RA_CERT_BLOB=$(sed -rn '/^-----BEGIN CERTIFICATE-----$/{:1;n;/^-----END CERTIFICATE-----$/b2;H;b1};:2;${x;s/\s//g;p}' /tmp/ra-agent.pem)

      2. Extract the serial number from the new certificate and store it in a variable:

         # RA_CERT_SERIAL=$(openssl x509 -in /tmp/ra-agent.pem -noout -serial | cut -d'=' -f2)

      3. Ensure the IdM services are running before attempting to modify the LDAP database:

         # ipactl start -f

      4. Update the LDAP entry for the RA agent with the new certificate blob and serial number. You will be prompted for the Directory Manager password:

         # ldapmodify -D "cn=Directory Manager" -W -x <<EOF
         dn: uid=ipara,ou=people,o=ipaca
         changetype: modify
         add: userCertificate
         userCertificate:: ${RA_CERT_BLOB}
         
         replace: description
         description: 2;${RA_CERT_SERIAL};CN=Certificate Authority,O=IDM.EXAMPLE.COM;CN=IPA RA,O=IDM.EXAMPLE.COM
         EOF

         Replace IDM.EXAMPLE.COM with your IdM realm or use your custom certificate subject base.

6. On the failing replica, replace the expired subsystem certificate.

   1. Import the healthy certificate into the NSS database. This requires the NSS database password, which can be found in /etc/pki/pki-tomcat/password.conf.

      1. Set a variable for the location of the NSS database password file for easier use in the following commands:

         # PWDFILE=/etc/pki/pki-tomcat/alias/pwdfile.txt

      2. Remove the old, expired subsystem certificate from the NSS database:

         # certutil -D -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -f $PWDFILE

      3. Add the new, healthy subsystem certificate from the /tmp/subsystem.pem file to the NSS database:

         # certutil -A -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -t ",," -i /tmp/subsystem.pem -f $PWDFILE

   2. Update the corresponding entry in LDAP, similar to the RA agent step:

      # SUBSYS_CERT_BLOB=$(cat /tmp/subsystem.pem | sed -rn '/^-----BEGIN CERTIFICATE-----$/{:1;n;/^-----END CERTIFICATE-----$/b2;H;b1};:2;${x;s/\s//g;p}')
      
      # SUBSYS_CERT_SERIAL=$(openssl x509 -in /tmp/subsystem.pem -noout -serial | cut -d'=' -f2)
      
      # ldapmodify -D "cn=Directory Manager" -W -x <<EOF
      dn: uid=pkidbuser,ou=people,o=ipaca
      changetype: modify
      add: userCertificate
      userCertificate:: ${SUBSYS_CERT_BLOB}
      
      replace: description
      description: 2;${SUBSYS_CERT_SERIAL};CN=Certificate Authority,O=IDM.EXAMPLE.COM;CN=CA Subsystem,O=IDM.EXAMPLE.COM
      EOF

      Replace IDM.EXAMPLE.COM with your IdM realm or use your custom certificate subject base.

7. Manually issue a temporary certificate for the replica’s LDAP service.

   1. On the failing replica, generate a Certificate Signing Request (CSR) from the existing key in the Directory Server’s NSS database:

      # DS_INSTANCE=$(grep ldap_uri /etc/ipa/default.conf | sed -n 's|.%2Frun%2F\(slapd-\)\.socket.|\1|p')*
      
      # LDAP_CERT_NICKNAME=$(grep nsSSLPersonalitySSL "/etc/dirsrv/${DS_INSTANCE}/dse.ldif" | awk '{ print $2 }')
      
      # HOSTNAME=$(hostname -f)
      
      # PWDFILE="/etc/dirsrv/${DS_INSTANCE}/pwdfile.txt"
      
      # certutil -R -d "/etc/dirsrv/${DS_INSTANCE}" -k ${LDAP_CERT_NICKNAME} -n ${LDAP_CERT_NICKNAME} -s "CN=${HOSTNAME}" -a -f "${PWDFILE}" -8 "${HOSTNAME}" -o /tmp/ldap.csr

      참고

      The command above creates /tmp/ldap.csr. Some versions of certutil may add extra text to the file. Ensure the file contains only the -----BEGIN NEW CERTIFICATE REQUEST----- block and nothing else.

   2. If the certificate is signed by the IdM CA:

      1. On the failing replica, copy the CSR to the renewal server:

         # scp /tmp/ldap.csr root@renewal-master.idm.example.com:/tmp/

      2. On the renewal server, sign the CSR. Use the ldap/ principal to ensure the certificate is valid for GSSAPI binds:

         # ipa cert-request /tmp/ldap.csr --principal="ldap/$(hostname -f)" --certificate-out=/tmp/ldap.pem

      3. On the renewal server, copy the new certificate back to the replica:

         # scp /tmp/ldap.pem root@failed-replica.idm.example.com:/tmp/

      4. On the failing replica, import the new certificate into the Directory Server’s NSS database:

         # certutil -D -d "/etc/dirsrv/${DS_INSTANCE}" -n ${LDAP_CERT_NICKNAME} -f "${PWDFILE}"
         
         # certutil -A -d "/etc/dirsrv/${DS_INSTANCE}" -n ${LDAP_CERT_NICKNAME} -t ",," -i /tmp/ldap.pem -f "${PWDFILE}"

   3. If the certificate is signed by an external CA:

      1. Take the /tmp/ldap.csr file and submit it to your external Certificate Authority for signing.
      2. Once you receive the new certificate file (for example, new_ldap_cert.pem), copy it securely to the /tmp/ directory on the failing replica and rename it to ldap.pem.

      3. On the failing replica, import the new certificate into the Directory Server’s NSS database:

         # certutil -D -d "/etc/dirsrv/${DS_INSTANCE}" -n ${ LDAP_CERT_NICKNAME} -f "${PWDFILE}"
         
         # certutil -A -d "/etc/dirsrv/${DS_INSTANCE}" -n ${ LDAP_CERT_NICKNAME} -t ",," -i /tmp/ldap.pem -f "${PWDFILE}"

8. Manually issue a temporary certificate for the replica’s PKI Tomcat service.

   1. On the affected replica, generate a CSR using the existing key stored in the PKI Tomcat NSS database:

      # HOSTNAME=$(hostname -f)
      
      # PWDFILE="/etc/pki/pki-tomcat/alias/pwdfile.txt"
      
      # certutil -R -d /etc/pki/pki-tomcat/alias -k 'Server-Cert cert-pki-ca' -n 'Server-Cert cert-pki-ca' -s "CN=${HOSTNAME},O=IDM.EXAMPLE.COM" -a -f "${PWDFILE}" -o /tmp/server-cert.csr

      Replace IDM.EXAMPLE.COM with your IdM realm or use your custom certificate subject base.

   2. On the failing replica, copy the CSR to the renewal server:

      # scp /tmp/server-cert.csr root@renewal-master.idm.example.com:/tmp/

   3. On the renewal server, sign the CSR. Use the host/ principal for the web server certificate:

      # ipa cert-request /tmp/server-cert.csr --principal="host/$(hostname -f)" --certificate-out=/tmp/server-cert.pem

   4. On the renewal server, copy the new certificate back to the replica:

      # scp /tmp/server-cert.pem root@failed-replica.idm.example.com:/tmp/

   5. On the failing replica, import the new certificate into the PKI Tomcat NSS database:

      # certutil -D -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' -f "${PWDFILE}"
      
      # certutil -A -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' -t ",," -i /tmp/server-cert.pem -f "${PWDFILE}"

9. Restart services and renew remaining certificates.

   1. Restart the IdM services on the replica to use the new certificates:

      # ipactl restart

   2. Restart the certmonger service to apply the manual changes. Once restarted, it should begin attempting to renew any remaining certificates:

      # systemctl restart certmonger

   3. You can watch the renewal process by periodically running getcert list | grep -E "Request ID|status|expires". Once all certificates are in the MONITORING state, the process is complete.

   4. The certmonger service should now be able to communicate with the CA. If one or more certificates remain unrenewed after a few minutes, consider renewing them manually:

      # getcert list
      
      # getcert resubmit -i <REQUEST_ID>