Appendix A. Troubleshooting
A.1. Troubleshooting SSSD
A.1.1. Setting Debug Logs for SSSD Domains
Each domain sets its own debug log level. Increasing the log level can provide more information about problems with SSSD or with the domain configuration.
To change the log level, set the
debug_level parameter for each section in the sssd.conf file for which to produce extra logs. For example:
[domain/LDAP]
cache_credentials = true
debug_level = 9| Level | Description |
|---|---|
| 0 | Fatal failures. Anything that would prevent SSSD from starting up or causes it to cease running. |
| 1 | Critical failures. An error that does not kill the SSSD, but one that indicates that at least one major feature is not going to work properly. |
| 2 | Serious failures. An error announcing that a particular request or operation has failed. |
| 3 | Minor failures. These are the errors that would percolate down to cause the operation failure of 2. |
| 4 | Configuration settings. |
| 5 | Function data. |
| 6 | Trace messages for operation functions. |
| 7 | Trace messages for internal control functions. |
| 8 | Contents of function-internal variables that may be interesting. |
| 9 | Extremely low-level tracing information. |
To change the debug level while SSSD is running, use the
sss_debuglevel utility, which is part of the sssd-tools package. For more information about how it works, see the sss_debuglevel man page.
A.1.2. Checking SSSD Log Files
SSSD uses a number of log files to report information about its operation, located in the
/var/log/sssd/ directory. SSSD produces a log file for each domain, as well as an sssd_pam.log and an sssd_nss.log file.
krb5_child.log: log file for the short-lived helper process involved in Kerberos authenticationldap_child.log: log file for the short-lived helper process involved in communicating with the LDAP serverselinux_child.log: log file for the short-lived helper process that retrieves SELinux informationsssd.log: log file for SSSD communicating with its responder processessssd_[domain].log: each SSSD domain section logs information about communication with the LDAP server to a separate log filesssd_ifp.log: log file for the InfoPipe responder, provides a public D-Bus interface accessible over the system bussssd_nss.log: log file for the Name Services Switch (NSS) responder that retrieves user and group informationsssd_pac.log: log file for the Microsoft Privilege Attribute Certificate (PAC) responder service that defines how SSSD works with Kerberos to manage Active Directory users and groupssssd_pam.log: log file for the Pluggable Authentication Module (PAM) respondersssd_ssh.log: log file for the SSH responder process
Additionally, the
/var/log/secure file logs authentication failures and the reason for the failure.
A.1.3. Problems with SSSD Configuration
- Q: SSSD fails to start
- Q: I do not see any groups with id or group members with getent group.
- Q: Authentication fails against LDAP.
- Q: Connecting to LDAP servers on non-standard ports fail.
- Q: NSS fails to return user information
- Q: NSS returns incorrect user information
- Q: Setting the password for the local SSSD user prompts twice for the password
- Q: An Active Directory identity provider is properly configured in my sssd.conf file, but SSSD fails to connect to it, with GSS-API errors.
- Q: I configured SSSD for central authentication, but now several of my applications (such as Firefox or Adobe) will not start.
- Q: SSSD is showing an automount location that I removed.
Q:
SSSD fails to start
A:
SSSD requires that the configuration file be properly set up, with all the required entries, before the daemon will start.
- SSSD requires at least one properly configured domain before the service will start. Without a domain, attempting to start SSSD returns an error that no domains are configured:
# sssd -d4 -i [sssd] [ldb] (3): server_sort:Unable to register control with rootdse! [sssd] [confdb_get_domains] (0): No domains configured, fatal error! [sssd] [get_monitor_config] (0): No domains configured.
Edit the/etc/sssd/sssd.conffile and create at least one domain. - SSSD also requires at least one available service provider before it will start. If the problem is with the service provider configuration, the error message indicates that there are no services configured:
[sssd] [get_monitor_config] (0): No services configured!
Edit the/etc/sssd/sssd.conffile and configure at least one service provider.Important
SSSD requires that service providers be configured as a comma-separated list in a singleservicesentry in the/etc/sssd/sssd.conffile. If services are listed in multiple entries, only the last entry is recognized by SSSD. - SSSD also requires the ownership and permissions of the
/etc/sssd/sssd.confto be set correctly. If the ownership or permissions are set incorrectly, attempt to start SSSD returns these error messages:[sssd] [confdb_ldif_from_ini_file] (0x0020): Permission check on config file failed. [sssd] [confdb_init_db] (0x0020): Cannot convert INI to LDIF [1]: [Operation not permitted] [sssd] [confdb_setup] (0x0010): ConfDB initialization has failed [1]: Operation not permitted [sssd] [load_configuration] (0x0010): Unable to setup ConfDB [1]: Operation not permitted [sssd] [main] (0x0020): Cannot read config file /etc/sssd/sssd.conf. Please check that the file is accessible only by the owner and owned by root.root.
Set the correct ownership and permissions of the/etc/sssd/sssd.conffile:# chmod 600 /etc/sssd/sssd.conf # chown root:root /etc/sssd/sssd.conf
Q:
I do not see any groups with
id or group members with getent group.
A:
This may be due to an incorrect
ldap_schema setting in the [domain/DOMAINNAME] section of sssd.conf.
SSSD supports RFC 2307 and RFC 2307bis schema types. By default, SSSD uses the more common RFC 2307 schema.
The difference between RFC 2307 and RFC 2307bis is the way which group membership is stored in the LDAP server. In an RFC 2307 server, group members are stored as the multi-valued
memberuid attribute, which contains the name of the users that are members. In an RFC2307bis server, group members are stored as the multi-valued member or uniqueMember attribute which contains the DN of the user or group that is a member of this group. RFC2307bis allows nested groups to be maintained as well.
If group lookups are not returning any information:
- Set
ldap_schematorfc2307bis. - Delete
/var/lib/sss/db/cache_DOMAINNAME.ldb. - Restarting SSSD.
If that does not work, add this line to
sssd.conf:
ldap_group_name = uniqueMember
Then delete the cache and restart SSSD again.
Q:
Authentication fails against LDAP.
A:
To perform authentication, SSSD requires that the communication channel be encrypted. This means that if
sssd.conf is configured to connect over a standard protocol (ldap://), it attempts to encrypt the communication channel with Start TLS. If sssd.conf is configured to connect over a secure protocol (ldaps://), then SSSD uses SSL.
This means that the LDAP server must be configured to run in SSL or TLS. TLS must be enabled for the standard LDAP port (389) or SSL enabled on the secure LDAPS port (636). With either SSL or TLS, the LDAP server must also be configured with a valid certificate trust.
An invalid certificate trust is one of the most common issues with authenticating against LDAP. If the client does not have proper trust of the LDAP server certificate, it is unable to validate the connection, and SSSD refuses to send the password. The LDAP protocol requires that the password be sent in plain text to the LDAP server. Sending the password in plain text over an unencrypted connection is a security problem.
If the certificate is not trusted, a
syslog message is written, indicating that TLS encryption could not be started. The certificate configuration can be tested by checking if the LDAP server is accessible apart from SSSD. For example, this tests an anonymous bind over a TLS connection to test.example.com:
$ ldapsearch -x -ZZ -h test.example.com -b dc=example,dc=com
If the certificate trust is not properly configured, the test fails with this error:
ldap_start_tls: Connect error (-11) additional info: TLS error -8179:Unknown code ___f 13
To trust the certificate:
- Obtain a copy of the public CA certificate for the certificate authority used to sign the LDAP server certificate and save it to the local system.
- Add a line to the
sssd.conffile that points to the CA certificate on the filesystem.ldap_tls_cacert = /path/to/cacert
- If the LDAP server uses a self-signed certificate, remove the
ldap_tls_reqcertline from thesssd.conffile.This parameter directs SSSD to trust any certificate issued by the CA certificate, which is a security risk with a self-signed CA certificate.
Q:
Connecting to LDAP servers on non-standard ports fail.
A:
When running SELinux in enforcing mode, the client's SELinux policy has to be modified to connect to the LDAP server over the non-standard port. For example:
# semanage port -a -t ldap_port_t -p tcp 1389
Q:
NSS fails to return user information
A:
This usually means that SSSD cannot connect to the NSS service.
- Ensure that the NSS service is running:
# service sssd status Redirecting to /bin/systemctl status sssd.service sssd.service - System Security Services Daemon Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled) Active: active (running) since Wed 2015-01-14 10:17:26 CET; 1min 30s ago Process: 683 ExecStart=/usr/sbin/sssd -D -f (code=exited, status=0/SUCCESS) Main PID: 745 (sssd) CGroup: /system.slice/sssd.service ├─745 /usr/sbin/sssd -D -f ├─746 /usr/libexec/sssd/sssd_be --domain default --debug-to-files... ├─804 /usr/libexec/sssd/sssd_nss --debug-to-files └─805 /usr/libexec/sssd/sssd_pam --debug-to-filesNSS service is running when SSSD is in theActive: active (running)state and when the output includessssd_nss. - If NSS is running, make sure that the provider is properly configured in the
[nss]section of the/etc/sssd/sssd.conffile. Especially check thefilter_usersandfilter_groupsattributes. - Make sure that NSS is included in the list of services that SSSD uses.
- Check the configuration in the
/etc/nsswitch.conffile. For more information, see the section called “Configure NSS Services to Use SSSD”.
Q:
NSS returns incorrect user information
A:
If searches are returning the incorrect user information, check that there are not conflicting user names in separate domains. When there are multiple domains, set the
use_fully_qualified_domains attribute to true in the /etc/sssd/sssd.conf file. This differentiates between different users in different domains with the same name.
Q:
Setting the password for the local SSSD user prompts twice for the password
A:
When attempting to change a local SSSD user's password, it may prompt for the password twice:
[root@clientF11 tmp]# passwd user1000 Changing password for user user1000. New password: Retype new password: New Password: Reenter new Password: passwd: all authentication tokens updated successfully.
This is the result of an incorrect PAM configuration. Ensure that the
use_authtok option is correctly configured in your /etc/pam.d/system-auth file. For examples of the correct configuration, see Section 7.5.2, “Configuring Services: PAM”.
Q:
An Active Directory identity provider is properly configured in my
sssd.conf file, but SSSD fails to connect to it, with GSS-API errors.
A:
SSSD can only connect with an Active Directory provider using its host name. If the host name is not given, the SSSD client cannot resolve the IP address to the host, and authentication fails.
For example, with this configuration:
[domain/ADEXAMPLE]
debug_level = 0xFFF0
id_provider = ad
ad_server = 172.16.0.1
ad_domain = example.com
krb5_canonicalize = False
The SSSD client returns this GSS-API failure, and the authentication request fails:
(Fri Jul 27 18:27:44 2012) [sssd[be[ADTEST]]] [sasl_bind_send] (0x0020): ldap_sasl_bind failed (-2)[Local error] (Fri Jul 27 18:27:44 2012) [sssd[be[ADTEST]]] [sasl_bind_send] (0x0080): Extended failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Cannot determine realm for numeric host address)]
To avoid this error, set the
ad_server to the name of the Active Directory host, or use the _srv_ keyword to use the DNS service discovery, as described in Section 7.4.3, “Configuring DNS Service Discovery”.
Q:
I configured SSSD for central authentication, but now several of my applications (such as Firefox or Adobe) will not start.
A:
Even on 64-bit systems, 32-bit applications require a 32-bit version of SSSD client libraries to use to access the password and identity cache. If a 32-bit version of SSSD is not available, but the system is configured to use the SSSD cache, then 32-bit applications can fail to start.
For example, Firefox can fail with permission denied errors:
Failed to contact configuration server. See http://www.gnome.org/projects/gconf/ for information. (Details - 1: IOR file '/tmp/gconfd-somebody/lock/ior' not opened successfully, no gconfd located: Permission denied 2: IOR file '/tmp/gconfd-somebody/lock/ior' not opened successfully, no gconfd located: Permission denied)
For Adobe Reader, the error shows that the current system user is not recognized:
[jsmith@server ~]$ acroread (acroread:12739): GLib-WARNING **: getpwuid_r(): failed due to unknown user id (366)
Other applications may show similar user or permissions errors.
Q:
SSSD is showing an automount location that I removed.
A:
The SSSD cache for the automount location persists even if the location is subsequently changed or removed. To update the autofs information in SSSD:
- Remove the autofs cache, as described in Section A.1.4, “A User Cannot Log In After UID or GID Changed”.
- Restart SSSD:
# systemctl restart sssd
A.1.4. A User Cannot Log In After UID or GID Changed
After a user or group ID changed, SSSD prevents a user from logging in.
What this means:
SSSD recognizes users and groups based on user IDs (UID) and group IDs (GID). When the UID or GID of an existing user or group changes, SSSD does not recognize the user or group.
To fix the problem:
Clear the SSSD cache using the
sss_cache utility:
- Make sure the sssd-tools is installed.
- To clear the SSSD cache for a specific user and leave the rest of the cache records intact:
# sss_cache --user user_name
To clear the cache for an entire domain:# sss_cache --domain domain_name
The utility invalidates records in the SSSD cache for a user, group, or domain. After this, SSSD retrieves the records from the identity provider to refresh the cache.
For details on
sss_cache, see the sss_cache(8) man page.
A.1.5. SSSD Control and Status Utility
SSSD provides the
sssctl utility to obtain status information, manage data files during troubleshooting, and other SSSD-related tasks.
- To use
sssctl, install the sssd-tools package:[root@server ~]# yum install sssd-tools
- Some options of
sssctluse the SSSD InfoPipe responder. To enable it, addifpto theservicesoption of your/etc/sssd/sssd.conf:[sssd] services = nss, sudo, pam, ssh, ifp - Restart SSSD:
[root@server ~]# systemctl restart sssd.service
A.1.5.1. SSSD Configuration Validation
The
sssctl config-check command performs a static analysis of the SSSD configuration files. This enables you to validate the /etc/sssd/sssd.conf file and /etc/sssd/conf.d/* files to receive a report before restarting SSSD.
The command performs the following checks on SSSD configuration files:
- Permissions
- The owner and group owner must be set to
root:rootand the permissions to600. - File names
- File names in
/etc/sssd/conf.d/must use the suffix.confand not start with a period (hidden files). - Typographical errors
- Typographical errors are checked in section and option names. Note that values are not checked.
- Options
- Options must be placed in the correct sections.
To verify the configuration, run:
# sssctl config-check Issues identified by validators: 3 [rule/allowed_sections]: Section [paM] is not allowed. Check for typos. [rule/allowed_domain_options]: Attribute 'offline_timeoutX' is not allowed in section 'domain/idm.example.com'. Check for typos. [rule/allowed_sudo_options]: Attribute 'homedir_substring' is not allowed in section 'sudo'. Check for typos. Messages generated during configuration merging: 2 File /etc/sssd/conf.d/wrong-file-permissions.conf did not pass access check. Skipping. File configuration.conf.disabled did not match provided patterns. Skipping. Used configuration snippet files: 1 /etc/sssd/conf.d/sample.conf
A.1.5.2. Displaying User Data
The
sssctl user-checks command helps to debug problems in applications that use SSSD as a back end for user lookup, authentication, and authorization. The command displays user data available through NSS and the InfoPipe responder for the D-Bus interface. The displayed data shows whether the user is authorized to log in using the system-auth PAM services.
For example, to display the user data of the
admin user:
# sssctl user-checks admin user: admin action: acct service: system-auth SSSD nss user lookup result: - user name: admin - user id: 1194200000 - group id: 1194200000 - gecos: Administrator - home directory: /home/admin - shell: /bin/bash SSSD InfoPipe user lookup result: - name: admin - uidNumber: 1194200000 - gidNumber: 1194200000 - gecos: Administrator - homeDirectory: /home/admin - loginShell: /bin/bash testing pam_acct_mgmt pam_acct_mgmt: Success PAM Environment: - no env -
For further options, see the output of the
sssctl user-checks --help command.
A.1.5.3. Domain Information
The domain status displays a list of domains SSSD accesses, and enables you to retrieve their status.
- List all domains available within SSSD, including trusted domains:
[root@server ~]# sssctl domain-list idm.example.com ad.example.com
- Retrieve the status of the domain idm.example.com:
[root@server ~]# sssctl domain-status idm.example.com Online status: Online
A.1.5.4. Cached Entries Information
The
sssctl command enables you to retrieve information about a cached entry, to investigate and debug cache-related authentication problems.
To query cache information for the user account
admin, run:
[root@server ~]# sssctl user-show admin Name: admin Cache entry creation date: 07/10/16 16:09:18 Cache entry last update time: 07/14/16 10:13:32 Cache entry expiration time: 07/14/16 11:43:32 Initgroups expiration time: Expired Cached in InfoPipe: No
To query the cache information for a group or netgroup, use:
[root@server ~]# sssctl group-show groupname [root@server ~]# sssctl netgroup-show netgroupname
A.1.5.5. Truncating the Log Files
When you debug a problem, you can use
sssctl logs-remove to truncate all SSSD log files in the /var/log/sssd/ directory to capture only newly created entries.
[root@server ~]# sssctl logs-remove Truncating log files...
A.1.5.6. Removing the SSSD Cache
To remove the SSSD cache database files, the
sssctl command provides the remove-cache option. Before the databases are removed, the command creates automatically a backup.
Use the following command to back up all local data and remove the SSSD cache:
[root@server ~]# sssctl cache-remove SSSD must not be running. Stop SSSD now? (yes/no) [yes] Creating backup of local data... Removing cache files... SSSD needs to be running. Start SSSD now? (yes/no) [yes]
Note
The backup stores only local data, such as local overrides, in the
/var/lib/sss/backup/ directory.
To automatically import the backed-up content, run
sssctl restore-local-data.
A.1.5.7. Obtaining Information about an LDAP Group Takes Long
Operations that involve looking up information about an LDAP group take very long, especially in case of large groups or nested groups.
What this means:
By default, LDAP group information lookups return all members for the group. For operations that involve large groups or nested groups, returning all members makes the process longer.
To fix the problem:
The membership lists returned in group lookups are not used when evaluating whether a user belongs to a group. To improve performance, especially for identity lookups, disable the group membership lookup:
- Open the
/etc/sssd/sssd.conffile. - In the
[domain]section, set theignore_group_membersoption totrue.[domain/domain_name] [... file truncated ...]
ignore_group_members = true
Note
This option should not be set to
true when the deployment involves an IdM server with the compat tree.
