Chapter 13. Troubleshooting authentication with SSSD in IdM

1. The getent command triggers the getpwnam call from the libc library.
2. The libc library references the /etc/nsswitch.conf configuration file to check which service is responsible for providing user information, and discovers the entry sss for the SSSD service.
3. The libc library opens the nss_sss module.
4. The nss_sss module checks the memory-mapped cache for the user information. If the data is present in the cache, the nss_sss module returns it.
5. If the user information is not in the memory-mapped cache, the request is passed to the SSSD sssd_nss responder process.
6. The SSSD service checks its cache. If the data is present in the cache and valid, the sssd_nss responder reads the data from the cache and returns it to the application.
7. If the data is not present in the cache or it is expired, the sssd_nss responder queries the appropriate back-end process and waits for a reply. The SSSD service uses the IPA backend in an IdM environment, enabled by the setting id_provider=ipa in the sssd.conf configuration file.
8. The sssd_be back-end process connects to the IdM server and requests the information from the IdM LDAP Directory Server.
9. The SSSD back-end on the IdM server responds to the SSSD back-end process on the IdM client.
10. The SSSD back-end on the client stores the resulting data in the SSSD cache and alerts the responder process that the cache has been updated.
11. The sssd_nss front-end responder process retrieves the information from the SSSD cache.
12. The sssd_nss responder sends the user information to the nss_sss responder, completing the request.
13. The libc library returns the user information to the application that requested it.

1. The IdM client looks to its local SSSD cache for AD user information.
2. If the IdM client does not have the user information, or the information is stale, the SSSD service on the client contacts the extdom_extop plugin on the IdM server to perform an LDAP extended operation and requests the information.
3. The SSSD service on the IdM server looks for the AD user information in its local cache.
4. If the IdM server does not have the user information in its SSSD cache, or its information is stale, it performs an LDAP search to request the user information from an AD Domain Controller.
5. The SSSD service on the IdM server receives the AD user information from the AD domain controller and stores it in its cache.
6. The extdom_extop plugin receives the information from the SSSD service on the IdM server, which completes the LDAP extended operation.
7. The SSSD service on the IdM client receives the AD user information from the LDAP extended operation.
8. The IdM client stores the AD user information in its SSSD cache and returns the information to the application that requested it.

1. The authentication attempt with the ssh command triggers the libpam library.

2. The libpam library references the PAM file in the /etc/pam.d/ directory that corresponds to the service requesting the authentication attempt. In this example involving authenticating via the SSH service on the local host, the libpam library checks the /etc/pam.d/system-auth configuration file and discovers the pam_sss.so entry for the SSSD PAM:

   auth    sufficient    pam_sss.so

3. To determine which authentication methods are available, the libpam library opens the pam_sss module and sends an SSS_PAM_PREAUTH request to the sssd_pam PAM responder of the SSSD service.
4. If smart card authentication is configured, the SSSD service spawns a temporary p11_child process to check for a smart card and retrieve certificates from it.
5. If smart card authentication is configured for the user, the sssd_pam responder attempts to match the certificate from the smart card with the user. The sssd_pam responder also performs a search for the groups that the user belongs to, since group membership might affect access control.
6. The sssd_pam responder sends an SSS_PAM_PREAUTH request to the sssd_be back-end responder to see which authentication methods the server supports, such as passwords or 2-factor authentication. In an IdM environment, where the SSSD service uses the IPA responder, the default authentication method is Kerberos. For this example, the user authenticates with a simple Kerberos password.
7. The sssd_be responder spawns a temporary krb5_child process.
8. The krb5_child process contacts the KDC on the IdM server and checks for available authentication methods.

9. The KDC responds to the request:

   1. The krb5_child process evaluates the reply and sends the results back to the sssd_be backend process.
   2. The sssd_be backend process receives the result.
   3. The sssd_pam responder receives the result.
   4. The pam_sss module receives the result.
10. If password authentication is configured for the user, the pam_sss module prompts the user for their password. If smart card authentication is configured, the pam_sss module prompts the user for their smart card PIN.

11. The module sends an SSS_PAM_AUTHENTICATE request with the user name and password, which travels to:

    1. The sssd_pam responder.
    2. The sssd_be back-end process.
12. The sssd_be process spawns a temporary krb5_child process to contact the KDC.
13. The krb5_child process attempts to retrieve a Kerberos Ticket Granting Ticket (TGT) from the KDC with the user name and password the user provided.
14. The krb5_child process receives the result of the authentication attempt.

15. The krb5_child process:

    1. Stores the TGT in a credential cache.
    2. Returns the authentication result to the sssd_be back-end process.

16. The authentication result travels from the sssd_be process to:

    1. The sssd_pam responder.
    2. The pam_sss module.
17. The pam_sss module sets an environment variable with the location of the user’s TGT so other applications can reference it.

Procedure

1. Verify that the SSSD service and its processes are running.

   [root@client ~]# pstree -a | grep sssd
     |-sssd -i --logger=files
     |   |-sssd_be --domain implicit_files --uid 0 --gid 0 --logger=files
     |   |-sssd_be --domain example.com --uid 0 --gid 0 --logger=files
     |   |-sssd_ifp --uid 0 --gid 0 --logger=files
     |   |-sssd_nss --uid 0 --gid 0 --logger=files
     |   |-sssd_pac --uid 0 --gid 0 --logger=files
     |   |-sssd_pam --uid 0 --gid 0 --logger=files
     |   |-sssd_ssh --uid 0 --gid 0 --logger=files
     |   `-sssd_sudo --uid 0 --gid 0 --logger=files
     |-sssd_kcm --uid 0 --gid 0 --logger=files

2. Verify that the client can contact the user database server via the IP address.

   [user@client ~]$ ping <IP_address_of_the_database_server>

   If this step fails, check that your network and firewall settings allow direct communication between IdM clients and servers. See Using and configuring firewalld.

3. Verify that the client can discover and contact the IdM LDAP server (for IdM users) or AD domain controller (for AD users) via the fully qualified host name.

   [user@client ~]$ dig -t SRV _ldap._tcp.example.com @<name_server>
   [user@client ~]$ ping <fully_qualified_host_name_of_the_server>

   If this step fails, check your Dynamic Name Service (DNS) settings, including the /etc/resolv.conf file. See Configuring the order of DNS servers.

   Note

   By default, the SSSD service attempts to automatically discover LDAP servers and AD DCs through DNS service (SRV) records. Alternatively, you can restrict the SSSD service to use specific servers by setting the following options in the sssd.conf configuration file:

   • ipa_server = <fully_qualified_host_name_of_the_server>
   • ad_server = <fully_qualified_host_name_of_the_server>
   • ldap_uri = <fully_qualified_host_name_of_the_server>

   If you use these options, verify you can contact the servers listed in them.

4. Verify that the client can authenticate to the LDAP server and retrieve user information with ldapsearch commands.

   1. If your LDAP server is an IdM server, like server.example.com, retrieve a Kerberos ticket for the host and perform the database search authenticating with the host Kerberos principal:

      [user@client ~]$ kinit -k 'host/client.example.com@EXAMPLE.COM'
      [user@client ~]$ ldapsearch -LLL -Y GSSAPI -h server.example.com -b “dc=example,dc=com” uid=<user_name>

   2. If your LDAP server is an Active Directory (AD) Domain Controller (DC), like server.ad.example.com, retrieve a Kerberos ticket for the host and perform the database search authenticating with the host Kerberos principal:

      [user@client ~]$ kinit -k 'CLIENT$@AD.EXAMPLE.COM'
      [user@client ~]$ ldapsearch -LLL -Y GSSAPI -h server.ad.example.com -b “dc=example,dc=com” sAMAccountname=<user_name>

   3. If your LDAP server is a plain LDAP server, and you have set the ldap_default_bind_dn and ldap_default_authtok options in the sssd.conf file, authenticate as the same ldap_default_bind_dn account:

      [user@client ~]$ ldapsearch -xLLL -D "cn=ldap_default_bind_dn_value" -W -h ldapserver.example.com -b “dc=example,dc=com” uid=<user_name>

   If this step fails, verify that your database settings allow your host to search the LDAP server.

5. Since the SSSD service uses Kerberos encryption, verify you can obtain a Kerberos ticket as the user that is unable to log in.

   1. If your LDAP server is an IdM server:

      [user@client ~]$ kinit <user_name>

   2. If LDAP server database is an AD server:

      [user@client ~]$ kinit <user_name@AD.EXAMPLE.COM>

   If this step fails, verify that your Kerberos server is operating properly, all servers have their times synchronized, and that the user account is not locked.

6. Verify you can retrieve user information about the command line.

   [user@client ~]$ getent passwd <user_name>
   [user@client ~]$ id <user_name>

   If this step fails, verify that the SSSD service on the client can receive information from the user database:

   1. Review errors in the /var/log/messages log file.
   2. Enable detailed logging in the SSSD service, collect debugging logs, and review the logs for indications to the source of the issue.
   3. Optional: Open a Red Hat Technical Support case and provide the troubleshooting information you have gathered.

7. If you are allowed to run sudo on the host, use the sssctl utility to verify the user is allowed to log in.

   [user@client ~]$ sudo sssctl user-checks -a auth -s ssh <user_name>

   If this step fails, verify your authorization settings, such as your PAM configuration, IdM HBAC rules, and IdM RBAC rules:

   1. Ensure that the user’s UID is equal to or higher than UID_MIN, which is defined in the /etc/login.defs file.
   2. Review authorization errors in the /var/log/secure and /var/log/messages log files.
   3. Enable detailed logging in the SSSD service, collect debugging logs, and review the logs for indications to the source of the issue.
   4. Optional: Open a Red Hat Technical Support case and provide the troubleshooting information you have gathered.

Procedure

1. Open the /etc/sssd/sssd.conf file in a text editor.

2. Add the debug_level option to every section of the file, and set the debug level to the verbosity of your choice.

   [domain/example.com]
   debug_level = 6
   id_provider = ipa
   ...
   
   [sssd]
   debug_level = 6
   services = nss, pam, ifp, ssh, sudo
   domains = example.com
   
   [nss]
   debug_level = 6
   
   [pam]
   debug_level = 6
   
   [sudo]
   debug_level = 6
   
   [ssh]
   debug_level = 6
   
   [pac]
   debug_level = 6
   
   [ifp]
   debug_level = 6

3. Save and close the sssd.conf file.

4. Restart the SSSD service to load the new configuration settings.

   [root@server ~]# systemctl restart sssd

Procedure

1. Enable detailed SSSD debug logging on the IdM server.

   [root@server ~]# sssctl debug-level 6

2. Invalidate objects in the SSSD cache for the user that is experiencing authentication issues, so you do not bypass the LDAP server and retrieve information SSSD has already cached.

   [root@server ~]# sssctl cache-expire -u idmuser

3. Minimize the troubleshooting dataset by removing older SSSD logs.

   [root@server ~]# sssctl logs-remove

4. Attempt to switch to the user experiencing authentication problems, while gathering timestamps before and after the attempt. These timestamps further narrow the scope of the dataset.

   [root@server sssd]# date; su idmuser; date
   Mon Mar 29 15:33:48 EDT 2021
   su: user idmuser does not exist
   Mon Mar 29 15:33:49 EDT 2021

5. Optional: Lower the debug level if you do not wish to continue gathering detailed SSSD logs.

   [root@server ~]# sssctl debug-level 2

6. Review SSSD logs for information about the failed request. For example, reviewing the /var/log/sssd/sssd_example.com.log file shows that the SSSD service did not find the user in the cn=accounts,dc=example,dc=com LDAP subtree. This might indicate that the user does not exist, or exists in another location.

   (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [dp_get_account_info_send] (0x0200): Got request for [0x1][BE_REQ_USER][name=idmuser@example.com]
   ...
   (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(uid=idmuser)(objectclass=posixAccount)(uid=)(&(uidNumber=)(!(uidNumber=0))))][cn=accounts,dc=example,dc=com].
   (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sdap_get_generic_op_finished] (0x0400): Search result: Success(0), no errmsg set
   (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sdap_search_user_process] (0x0400): Search for users, returned 0 results.
   (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sysdb_search_by_name] (0x0400): No such entry
   (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sysdb_delete_user] (0x0400): Error: 2 (No such file or directory)
   (Mon Mar 29 15:33:48 2021) [sssd[be[example.com]]] [sysdb_search_by_name] (0x0400): No such entry
   (Mon Mar 29 15:33:49 2021) [sssd[be[example.com]]] [ipa_id_get_account_info_orig_done] (0x0080): Object not found, ending request

7. If you are unable to determine the cause of the authentication issue:

   1. Collect the SSSD logs you recently generated.

      [root@server ~]# sssctl logs-fetch sssd-logs-Mar29.tar

   2. Open a Red Hat Technical Support case and provide:

      1. The SSSD logs: sssd-logs-Mar29.tar

      2. The console output, including the time stamps and user name, of the request that corresponds to the logs:

         [root@server sssd]# date; id idmuser; date
         Mon Mar 29 15:33:48 EDT 2021
         id: ‘idmuser’: no such user
         Mon Mar 29 15:33:49 EDT 2021

Procedure

1. On the client: Open the /etc/sssd/sssd.conf file in a text editor.

2. On the client: Add the ipa_server option to the [domain] section of the file and set it to an IdM server. This avoids the IdM client autodiscovering other IdM servers, thus limiting this test to just one client and one server.

   [domain/example.com]
   ipa_server = server.example.com
   ...

3. On the client: Save and close the sssd.conf file.

4. On the client: Restart the SSSD service to load the configuration changes.

   [root@client ~]# systemctl restart sssd

5. On the server and client: Enable detailed SSSD debug logging.

   [root@server ~]# sssctl debug-level 6

   [root@client ~]# sssctl debug-level 6

6. On the server and client: Invalidate objects in the SSSD cache for the user experiencing authentication issues, so you do not bypass the LDAP database and retrieve information SSSD has already cached.

   [root@server ~]# sssctl cache-expire -u idmuser

   [root@client ~]# sssctl cache-expire -u idmuser

7. On the server and client: Minimize the troubleshooting dataset by removing older SSSD logs.

   [root@server ~]# sssctl logs-remove

   [root@server ~]# sssctl logs-remove

8. On the client: Attempt to switch to the user experiencing authentication problems while gathering timestamps before and after the attempt. These timestamps further narrow the scope of the dataset.

   [root@client sssd]# date; su idmuser; date
   Mon Mar 29 16:20:13 EDT 2021
   su: user idmuser does not exist
   Mon Mar 29 16:20:14 EDT 2021

9. Optional: On the server and client: Lower the debug level if you do not wish to continue gathering detailed SSSD logs.

   [root@server ~]# sssctl debug-level 0

   [root@client ~]# sssctl debug-level 0

10. On the server and client: Review SSSD logs for information about the failed request.

    1. Review the request from the client in the client logs.
    2. Review the request from the client in the server logs.
    3. Review the result of the request in the server logs.
    4. Review the outcome of the client receiving the results of the request from the server.

11. If you are unable to determine the cause of the authentication issue:

    1. Collect the SSSD logs you recently generated on the IdM server and IdM client. Label them according to their hostname or role.

       [root@server ~]# sssctl logs-fetch sssd-logs-server-Mar29.tar

       [root@client ~]# sssctl logs-fetch sssd-logs-client-Mar29.tar

    2. Open a Red Hat Technical Support case and provide:

       1. The SSSD debug logs:

          1. sssd-logs-server-Mar29.tar from the server
          2. sssd-logs-client-Mar29.tar from the client

       2. The console output, including the time stamps and user name, of the request that corresponds to the logs:

          [root@client sssd]# date; su idmuser; date
          Mon Mar 29 16:20:13 EDT 2021
          su: user idmuser does not exist
          Mon Mar 29 16:20:14 EDT 2021

Procedure

1. To review your SSSD log file, open the log file using the less utility. For example, to view the /var/log/sssd/sssd_example.com.log:

   [root@server ~]# less /var/log/sssd/sssd_example.com.log

2. Review the SSSD logs for information about the client request.

   (2021-07-26 18:26:37): [be[testidm.com]] [dp_req_destructor] (0x0400): [RID#3] Number of active DP request: 0
   (2021-07-26 18:26:37): [be[testidm.com]] [dp_req_reply_std] (0x1000): [RID#3] DP Request AccountDomain #3: Returning [Internal Error]: 3,1432158301,GetAccountDomain() not supported
   (2021-07-26 18:26:37): [be[testidm.com]] [dp_attach_req] (0x0400): [RID#4] DP Request Account #4: REQ_TRACE: New request. [sssd.nss CID #1] Flags [0x0001].
   (2021-07-26 18:26:37): [be[testidm.com]] [dp_attach_req] (0x0400): [RID#4] Number of active DP request: 1

   This sample output from an SSSD log file shows the unique identifiers RID#3 and RID#4 for two different requests.