Using IdM Healthcheck to monitor your IdM environment
Performing status and health checks
Abstract
ipa-healthcheck
utility helps administrators to detect problems in a Red Hat Identity Management (IdM) environment. This includes status checks of IdM services, configuration file permissions, replication statuses, and issues with certificates.
Providing feedback on Red Hat documentation
We appreciate your feedback on our documentation. Let us know how we can improve it.
Submitting feedback through Jira (account required)
- Log in to the Jira website.
- Click Create in the top navigation bar
- Enter a descriptive title in the Summary field.
- Enter your suggestion for improvement in the Description field. Include links to the relevant parts of the documentation.
- Click Create at the bottom of the dialogue.
Chapter 1. Installing and running the IdM Healthcheck tool
Learn more about the IdM Healthcheck tool and how to install and run it.
1.1. Healthcheck in IdM
The Healthcheck
command line tool in Identity Management (IdM) helps find issues that can impact the performance of your IdM environment. Using Healthcheck, you can identify an issue in advance so that you can correct it before it becomes critical.
You can use Healthcheck without obtaining a Kerberos ticket.
Modules are independent
Healthcheck consists of independent modules which check for:
- Replication issues
- Certificate validity
- Certificate authority infrastructure issues
- IdM and Active Directory trust issues
- Correct file permissions and ownership settings
Output formats and destination
You can set the following types of output for Healthcheck to generate by using the output-type
option:
-
json
: Machine-readable output in JSON format (default) -
human
: Human-readable output
You can specify a file to store the output by using the --output-file
option.
Results
Each Healthcheck module returns one of the following results:
- SUCCESS
- The system is configured as expected.
- WARNING
- It is advisable to monitor or evaluate the configuration.
- ERROR
- The system is not configured as expected.
- CRITICAL
- The configuration is not as expected, with a significant potential to impact the functioning of your IdM deployment.
Additional resources
-
man ipa-healthcheck
1.2. Installing IdM Healthcheck
Learn how you can install the IdM Healthcheck tool.
Prerequisites
-
You are logged in as
root
.
Procedure
Install the
ipa-healthcheck
package:Copy to Clipboard Copied! Toggle word wrap Toggle overflow dnf install ipa-healthcheck
[root@server ~]# dnf install ipa-healthcheck
Verification
Perform a basic Healthcheck test:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ipa-healthcheck
[root@server ~]# ipa-healthcheck []
The empty square brackets
[]
indicate a fully-functioning IdM installation.
Additional resources
-
Run
ipa-healthcheck --help
to see all supported arguments.
1.3. Running IdM Healthcheck
You can execute Healthcheck tests in one of the following ways:
- Manually
- Automatically by using log rotation.
This section describes how to execute the tests manually.
Prerequisites
- The Healthcheck tool is installed. See Installing IdM Healthcheck.
Procedure
[Optional] To display a list of all available Healthcheck tests, enter:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ipa-healthcheck --list-sources
[root@server ~]# ipa-healthcheck --list-sources
To run the Healthcheck utility, enter:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ipa-healthcheck
[root@server ~]# ipa-healthcheck
Additional resources
-
man ipa-healthcheck
1.4. Log rotation
Log rotation creates a new log file every day and the files are organized by date. The date is included in the filename.
Using log rotation, you can configure the maximum number of log files to be stored. If this number is exceeded, the newest file replaces the oldest one. For example, if the maximum rotation number is thirty, the thirty-first log file replaces the first, that is the oldest one.
Log rotation reduces voluminous log files and organizes them. This helps you analyze the logs.
1.5. Running IdM Healthcheck on a schedule
Follow this procedure to configure IdM Healthcheck to run on a schedule. This includes configuring the following utilities:
-
The
systemd
timer to run the Healthcheck tool periodically and generate the logs. -
The
crond
service to ensure log rotation.
The default log name is healthcheck.log
and the rotated logs use the healthcheck.log-YYYYMMDD
format.
The Healthcheck timer tool is not a real-time tool. It is only meant to be run a few times an hour. If you require real-time monitoring of, for example, services or disk space, use a different tool.
Prerequisites
-
You have
root
privileges.
Procedure
Enable a
systemd
timer:Copy to Clipboard Copied! Toggle word wrap Toggle overflow systemctl enable ipa-healthcheck.timer
# systemctl enable ipa-healthcheck.timer Created symlink /etc/systemd/system/multi-user.target.wants/ipa-healthcheck.timer -> /usr/lib/systemd/system/ipa-healthcheck.timer.
Start the
systemd
timer:Copy to Clipboard Copied! Toggle word wrap Toggle overflow systemctl start ipa-healthcheck.timer
# systemctl start ipa-healthcheck.timer
Open the
/etc/logrotate.d/ipahealthcheck
file to configure the number of logs you want to be saved:Copy to Clipboard Copied! Toggle word wrap Toggle overflow [...] rotate 30 }
[...] rotate 30 }
By default, logs are stored for 30 days before they are overwritten by newer logs.
In the same file, configure the path to the file storing the logs.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow /var/log/ipa/healthcheck/healthcheck.log { [...]
/var/log/ipa/healthcheck/healthcheck.log { [...]
By default, logs are saved in the
/var/log/ipa/healthcheck/
directory.- Save the file.
Ensure that the
crond
service is enabled and running:Copy to Clipboard Copied! Toggle word wrap Toggle overflow systemctl enable crond systemctl start crond
# systemctl enable crond # systemctl start crond
To start generating logs, start the IdM healthcheck service:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow systemctl start ipa-healthcheck
# systemctl start ipa-healthcheck
Verification
-
Navigate to the
/var/log/ipa/healthcheck/
directory. - Display the contents of the log file to check if it was created correctly.
1.6. IdM Healthcheck configuration modifications
You can change Identity Management (IdM) Healthcheck settings by adding the desired command line options to the /etc/ipahealthcheck/ipahealthcheck.conf
file. This can be useful when, for example, you configured log rotation previously and now want to ensure the logs are in a format suitable for automatic analysis, but do not want to set up a new timer.
This Healthcheck feature is only available on RHEL 9.1 and newer.
After the modification, all logs that Healthcheck creates follow the new settings. These settings also apply to any manual execution of Healthcheck.
When running Healthcheck manually, the settings in the configuration file take precedence over the options specified in the command line. For example, if output_type
is set to human
in the configuration file, specifying json
on the command line has no effect. Any command line options you use that are not specified in the configuration file are applied normally.
1.7. Configuring Healthcheck to change the output logs format
Follow this procedure to configure Healthcheck with a timer already configured. In this example, you re-configure Healthcheck to start producing logs in a human-readable format and to also include successful results instead of only errors.
Prerequisites
- Your system is running RHEL 9.1 or later.
-
You have
root
privileges. - You have previously configured log rotation on a timer.
Procedure
-
Open the
/etc/ipahealthcheck/ipahealthcheck.conf
file in a text editor. -
Add options
output_type=human
andall=True
to the[default]
section. - Save and close the file.
Verification
Run Healthcheck manually:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ipa-healthcheck
# ipa-healthcheck
-
Go to
/var/log/ipa/healthcheck/
and check that the logs are in the correct format.
1.8. Additional resources
See the following sections of the Using IdM Healthcheck to monitor your IdM environment guide for examples of using IdM Healthcheck.
Chapter 2. Checking services using IdM Healthcheck
You can monitor services used by the Identity Management (IdM) server using the Healthcheck tool.
2.1. The IdM services Healthcheck test
The Healthcheck tool includes a test to check if the Identity Management (IdM) services are running correctly. Start with this Healthcheck test as IdM services that are not running correctly can cause failures in other Healthcheck tests.
The services test is context-specific based on what features are configured. For example, named
is only checked if the integrated IdM DNS service is configured on the IdM server. Others, for example smb
or winbind
, are only checked if an IdM-AD trust is enabled.
The list of IdM services that the test evaluates can look as follows:
- certmonger
- dirsrv
- gssproxy
- httpd
- ipa_custodia
- ipa_otpd
- kadmin
- krb5kdc
- named
- ods_enforcerd
- ipa_dnskeysyncd
- pki_tomcatd
- sssd
- chronyd
- smb
- winbind
You can view this list by running the ipa-healthcheck --list-sources
command and identifying the ipahealthcheck.meta.services
section in the output.
2.2. Screening IdM services using Healthcheck
Follow this procedure to run a standalone manual test of services running on the Identity Management (IdM) server by using the Healthcheck tool.
Procedure
Enter:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ipa-healthcheck --source=ipahealthcheck.meta.services
# ipa-healthcheck --source=ipahealthcheck.meta.services
-
The
--source=ipahealthcheck.meta.services
option ensures that IdM Healthcheck only performs the services test. -
The
--failures-only
option is enabled by default and it ensures that IdM Healthcheck only reports warnings, errors and critical issues.
-
The
A successful test displays empty brackets:
[ ]
[ ]
If one of the services fails, the result can looks similarly to this example:
{ "source": "ipahealthcheck.meta.services", "check": "httpd", "result": "ERROR", "kw": { "status": false, "msg": "httpd: not running" } }
{
"source": "ipahealthcheck.meta.services",
"check": "httpd",
"result": "ERROR",
"kw": {
"status": false,
"msg": "httpd: not running"
}
}
Run this test on all IdM servers when trying to discover issues.
Additional resources
Chapter 3. Checking disk space using IdM Healthcheck
You can monitor the Identity Management server’s free disk space using the Healthcheck tool.
3.1. Disk space healthcheck test
The Healthcheck tool includes the FileSystemSpaceCheck
test for checking available disk space. The test checks the following:
- The minimum raw free bytes needed.
- The percentage — the minimum free disk space is hardcoded to 20%.
The test checks the following paths:
Paths checked by the test | Minimal disk space in MB |
---|---|
| 1024 |
| 512 |
| 1024 |
| 512 |
| 512 |
| 512 |
Insufficient free disk space can cause issues with the following:
- Logging
- Execution
- Backups
You can find the FileSystemSpaceCheck
test by running the ipa-healthcheck --list-sources
command and identifying the ipahealthcheck.system.filesystemspace
section in the output.
3.2. Screening disk space using the Healthcheck tool
Follow this procedure to run a standalone manual test of available disk space on an Identity Management (IdM) server using the Healthcheck tool.
Procedure
Enter:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ipa-healthcheck --source=ipahealthcheck.system.filesystemspace
# ipa-healthcheck --source=ipahealthcheck.system.filesystemspace
-
The
--source=ipahealthcheck.meta.services
option ensures that IdM Healthcheck only performs the disk space test.
-
The
A successful test displays empty brackets:
[]
[]
As an example, a failed test can display:
{ "source": "ipahealthcheck.system.filesystemspace", "check": "FileSystemSpaceCheck", "result": "ERROR", "kw": { "msg": "/var/lib/dirsrv: free space under threshold: 0 MiB < 1024 MiB", "store": "/var/lib/dirsrv", "free_space": 0, "threshold": 1024 } }
{
"source": "ipahealthcheck.system.filesystemspace",
"check": "FileSystemSpaceCheck",
"result": "ERROR",
"kw": {
"msg": "/var/lib/dirsrv: free space under threshold: 0 MiB < 1024 MiB",
"store": "/var/lib/dirsrv",
"free_space": 0,
"threshold": 1024
}
}
This failed test informs you that no space is available in the /var/lib/dirsrv
directory.
Run this test on all IdM servers when trying to discover issues.
Additional resources
Chapter 4. Verifying permissions of IdM configuration files using Healthcheck
Learn more about how to test Identity Management (IdM) configuration files using the Healthcheck tool.
4.1. File permissions Healthcheck tests
The Healthcheck tool tests the ownership and permissions of files installed or configured by Identity Management (IdM).
If you change the ownership or permissions of these files, the tests return a warning in the result
section. While this does not necessarily mean that the configuration does not work, it means that the file differs from the default configuration.
You can find the file permissions tests under the ipahealthcheck.ipa.files
source of the output of the ipa-healthcheck --list-sources
command.
- IPAFileNSSDBCheck
-
This test checks the 389-ds NSS database and the Certificate Authority (CA) database, if relevant. The 389-ds database is located in
/etc/dirsrv/slapd-<dashed-REALM>
and the CA database is located in/etc/pki/pki-tomcat/alias/
. - IPAFileCheck
This test checks the following files:
-
/var/lib/ipa/ra-agent.{key|pem}
-
/var/lib/ipa/certs/httpd.pem
-
/var/lib/ipa/private/httpd.key
-
/etc/httpd/alias/ipasession.key
-
/etc/dirsrv/ds.keytab
-
/etc/ipa/ca.crt
-
/etc/ipa/custodia/server.keys
-
/etc/resolv.conf
/etc/hosts
If PKINIT is enabled, it also tests:
-
/var/lib/ipa/certs/kdc.pem
/var/lib/ipa/private/kdc.key
If DNS is configured, it also tests:
-
/etc/named.keytab
-
/etc/ipa/dnssec/ipa-dnskeysyncd.keytab
-
- TomcatFileCheck
This test checks certain
tomcat
-specific files:-
/etc/pki/pki-tomcat/password.conf
-
/var/lib/pki/pki-tomcat/conf/ca/CS.cfg
-
/etc/pki/pki-tomcat/server.xml
-
4.2. Screening configuration files using Healthcheck
Follow this procedure to run a standalone manual test of an Identity Management (IdM) server’s configuration files using the Healthcheck tool.
The Healthcheck tool includes many tests. Results can be narrowed down by:
-
Including only ownership and permissions tests:
--source=ipahealthcheck.ipa.files
Procedure
To run Healthcheck tests on IdM configuration file ownership and permissions, while displaying only warnings, errors and critical issues, enter:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ipa-healthcheck --source=ipahealthcheck.ipa.files
# ipa-healthcheck --source=ipahealthcheck.ipa.files
A successful test displays empty brackets:
ipa-healthcheck --source=ipahealthcheck.ipa.files
# ipa-healthcheck --source=ipahealthcheck.ipa.files
[]
Failed tests display results similar to the following WARNING
:
{ "source": "ipahealthcheck.ipa.files", "check": "IPAFileNSSDBCheck", "result": "WARNING", "kw": { "key": "_etc_dirsrv_slapd-EXAMPLE-TEST_pkcs11.txt_mode", "path": "/etc/dirsrv/slapd-EXAMPLE-TEST/pkcs11.txt", "type": "mode", "expected": "0640", "got": "0666", "msg": "Permissions of /etc/dirsrv/slapd-EXAMPLE-TEST/pkcs11.txt are 0666 and should be 0640" } }
{
"source": "ipahealthcheck.ipa.files",
"check": "IPAFileNSSDBCheck",
"result": "WARNING",
"kw": {
"key": "_etc_dirsrv_slapd-EXAMPLE-TEST_pkcs11.txt_mode",
"path": "/etc/dirsrv/slapd-EXAMPLE-TEST/pkcs11.txt",
"type": "mode",
"expected": "0640",
"got": "0666",
"msg": "Permissions of /etc/dirsrv/slapd-EXAMPLE-TEST/pkcs11.txt are 0666 and should be 0640"
}
}
Run these tests on all IdM servers when trying to find issues.
Additional resources
Chapter 5. Checking DNS records using IdM Healthcheck
You can identify issues with DNS records in Identity Management (IdM) using the Healthcheck tool.
5.1. DNS records healthcheck test
The Healthcheck tool includes the IPADNSSystemRecordsCheck
test for checking that the expected DNS records required for autodiscovery are resolvable. Specifically, the test checks the DNS records obtained by the ipa dns-update-system-records --dry-run
command using the first resolver specified in the /etc/resolv.conf
file on the IdM server to which you are logged in.
You can find the IPADNSSystemRecordsCheck
test under the ipahealthcheck.ipa.idns
source of the output of the ipa-healthcheck --list-sources
command.
5.2. Screening IdM DNS records using the Healthcheck tool
Follow this procedure to run a standalone manual test of DNS records on an Identity Management (IdM) server using the Healthcheck tool.
The Healthcheck tool includes many tests. Results can be narrowed down by including only the DNS records tests by adding the --source ipahealthcheck.ipa.idns
option.
Prerequisites
-
You have
root
privileges.
Procedure
Enter:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ipa-healthcheck --source ipahealthcheck.ipa.idns
# ipa-healthcheck --source ipahealthcheck.ipa.idns
The
--source ipahealthcheck.ipa.idns
option ensures that IdM Healthcheck only performs the DNS records test.If the record is resolvable, the test returns
SUCCESS
as a result:Copy to Clipboard Copied! Toggle word wrap Toggle overflow { "source": "ipahealthcheck.ipa.idns", "check": "IPADNSSystemRecordsCheck", "result": "SUCCESS", "uuid": "eb7a3b68-f6b2-4631-af01-798cac0eb018", "when": "20200415143339Z", "duration": "0.210471", "kw": { "key": "_ldap._tcp.idm.example.com.:server1.idm.example.com." } }
{ "source": "ipahealthcheck.ipa.idns", "check": "IPADNSSystemRecordsCheck", "result": "SUCCESS", "uuid": "eb7a3b68-f6b2-4631-af01-798cac0eb018", "when": "20200415143339Z", "duration": "0.210471", "kw": { "key": "_ldap._tcp.idm.example.com.:server1.idm.example.com." } }
The test returns a
WARNING
when, for example, the number of records does not match the expected number:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow { "source": "ipahealthcheck.ipa.idns", "check": "IPADNSSystemRecordsCheck", "result": "WARNING", "uuid": "972b7782-1616-48e0-bd5c-49a80c257895", "when": "20200409100614Z", "duration": "0.203049", "kw": { "msg": "Got {count} ipa-ca A records, expected {expected}", "count": 2, "expected": 1 } }
{ "source": "ipahealthcheck.ipa.idns", "check": "IPADNSSystemRecordsCheck", "result": "WARNING", "uuid": "972b7782-1616-48e0-bd5c-49a80c257895", "when": "20200409100614Z", "duration": "0.203049", "kw": { "msg": "Got {count} ipa-ca A records, expected {expected}", "count": 2, "expected": 1 } }
Additional resources
Chapter 6. Verifying the optimal number of KDC worker processes using IdM Healthcheck
You can use the Healthcheck tool in Identity Management (IdM) to verify that the Kerberos Key Distribution Center (KDC) is configured to use the optimal number of krb5kdc
worker processes, which should be equal to the number of CPU cores on the host.
Prerequisites
- You are using RHEL 9.1 or later.
-
You have
root
privileges.
Procedure
Enter:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ipa-healthcheck --source ipahealthcheck.ipa.kdc
# ipa-healthcheck --source ipahealthcheck.ipa.kdc
The
--source ipahealthcheck.ipa.kdc
option ensures that IdM Healthcheck only performs theKDCWorkersCheck
test.If the number of KDC worker processes matches the number of CPU cores, the test returns
SUCCESS
as a result:Copy to Clipboard Copied! Toggle word wrap Toggle overflow { "source": "ipahealthcheck.ipa.kdc", "check": "KDCWorkersCheck", "result": "SUCCESS", "uuid": "68f6e20a-0aa9-427d-8fdc-fbb8196d56cd", "when": "20230105162211Z", "duration": "0.000157", "kw": { "key": "workers" } }
{ "source": "ipahealthcheck.ipa.kdc", "check": "KDCWorkersCheck", "result": "SUCCESS", "uuid": "68f6e20a-0aa9-427d-8fdc-fbb8196d56cd", "when": "20230105162211Z", "duration": "0.000157", "kw": { "key": "workers" } }
The test returns a
WARNING
if the number of worker processes does not match the number of CPU cores. In the following example, a host with 2 cores is configured to have only one KDC worker process:Copy to Clipboard Copied! Toggle word wrap Toggle overflow { "source": "ipahealthcheck.ipa.kdc", "check": "KDCWorkersCheck", "result": "WARNING", "uuid": "972b7782-1616-48e0-bd5c-49a80c257895", "when": "20230105122236Z", "duration": "0.203049", "kw": { "key": ‘workers’, "cpus": 2, "workers": 1, "expected": "The number of CPUs {cpus} does not match the number of workers {workers} in {sysconfig}" } }
{ "source": "ipahealthcheck.ipa.kdc", "check": "KDCWorkersCheck", "result": "WARNING", "uuid": "972b7782-1616-48e0-bd5c-49a80c257895", "when": "20230105122236Z", "duration": "0.203049", "kw": { "key": ‘workers’, "cpus": 2, "workers": 1, "expected": "The number of CPUs {cpus} does not match the number of workers {workers} in {sysconfig}" } }
The test also outputs a
WARNING
if there are no configured workers. In the following example, theKRB5KDC_ARGS
variable is missing from the/etc/sysconfig/krb5kdc
configuration file:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow { "source": "ipahealthcheck.ipa.kdc", "check": "KDCWorkersCheck", "result": "WARNING", "uuid": "5d63ea86-67b9-4638-a41e-b71f4 56efed7", "when": "20230105162526Z", "duration": "0.000135", "kw": { "key": "workers", "sysconfig": "/etc/sysconfig/krb5kdc", "msg": "KRB5KDC_ARGS is not set in {sysconfig}" } }
{ "source": "ipahealthcheck.ipa.kdc", "check": "KDCWorkersCheck", "result": "WARNING", "uuid": "5d63ea86-67b9-4638-a41e-b71f4 56efed7", "when": "20230105162526Z", "duration": "0.000135", "kw": { "key": "workers", "sysconfig": "/etc/sysconfig/krb5kdc", "msg": "KRB5KDC_ARGS is not set in {sysconfig}" } }
Additional resources
Chapter 7. Checking IdM replication using Healthcheck
You can test Identity Management (IdM) replication using the Healthcheck tool.
Prerequisites
- You are using RHEL version 8.1 or newer.
7.1. The IdM replication and topology Healthcheck tests
The Healthcheck tool includes tests of the Identity Management (IdM) topology configuration. The tests search for replication conflict issues.
You can find the IPATopologyDomainCheck
and ReplicationConflictCheck
tests under the ipahealthcheck.ipa.topology
and ipahealthcheck.ds.replication
sources of the output of the ipa-healthcheck --list-sources
command.
- IPATopologyDomainCheck
Tests the following configuration:
- No IdM server is disconnected from the topology.
- The IdM servers do not have more than the recommended number of replication agreements.
If the test succeeds, the test returns the configured domains. Otherwise, specific connection errors are reported.
NoteThe test runs the
ipa topologysuffix-verify
command for thedomain
suffix. It also runs the command for theca
suffix if the IdM Certificate Authority server role is configured on this server.- ReplicationConflictCheck
-
Searches for entries in LDAP matching
(&(!(objectclass=nstombstone))(nsds5ReplConflict=*))
.
7.2. Screening replication using Healthcheck
Follow this procedure to run a standalone manual test of your Identity Management (IdM) replication and topology configuration using the Healthcheck tool.
Prerequisites
-
You have
root
privileges.
Procedure
Enter:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ipa-healthcheck --source=ipahealthcheck.ds.replication --source=ipahealthcheck.ipa.topology
# ipa-healthcheck --source=ipahealthcheck.ds.replication --source=ipahealthcheck.ipa.topology
-
The
--source=ipahealthcheck.ds.replication
and--source=ipahealthcheck.ipa.topology
options ensure that IdM Healthcheck only performs the replication conflict and topology tests.
Four different results are possible:
SUCCESS — the test passed successfully.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow { "source": "ipahealthcheck.ipa.topology", "check": "IPATopologyDomainCheck", "result": "SUCCESS", "kw": { "suffix": "domain" } }
{ "source": "ipahealthcheck.ipa.topology", "check": "IPATopologyDomainCheck", "result": "SUCCESS", "kw": { "suffix": "domain" } }
- WARNING — the test passed but there might be a problem.
ERROR — the test failed.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow { "source": "ipahealthcheck.ipa.topology", "check": "IPATopologyDomainCheck", "result": "ERROR", "uuid": d6ce3332-92da-423d-9818-e79f49ed321f "when": 20191007115449Z "duration": 0.005943 "kw": { "msg": "topologysuffix-verify domain failed, server2 is not connected (server2_139664377356472 in MainThread)" } }
{ "source": "ipahealthcheck.ipa.topology", "check": "IPATopologyDomainCheck", "result": "ERROR", "uuid": d6ce3332-92da-423d-9818-e79f49ed321f "when": 20191007115449Z "duration": 0.005943 "kw": { "msg": "topologysuffix-verify domain failed, server2 is not connected (server2_139664377356472 in MainThread)" } }
- CRITICAL — the test failed and it affects the IdM server functionality.
-
The
Run these tests on all IdM servers when trying to check for issues.
Additional resources
7.3. Additional resources
Chapter 8. Verifying your IdM and AD trust configuration using IdM Healthcheck
Learn more about identifying issues with a trust between Identity Management (IdM) and Active Directory (AD) by using the Healthcheck tool.
8.1. IdM and AD trust Healthcheck tests
The Healthcheck tool includes several tests for testing the status of the trust between Identity Management (IdM) and Active Directory (AD).
To see all trust tests, run ipa-healthcheck
with the --list-sources
option:
ipa-healthcheck --list-sources
# ipa-healthcheck --list-sources
You can find all trust-related tests under the ipahealthcheck.ipa.trust
source:
- IPATrustAgentCheck
-
This test checks the SSSD configuration if the current host is configured as a trust agent. For each domain in
/etc/sssd/sssd.conf
whereid_provider=ipa
ensure thatipa_server_mode
isTrue
. - IPATrustDomainsCheck
-
This test checks if the trust domains match SSSD domains by comparing the list of domains in
sssctl domain-list
with the list of domains fromipa trust-find
excluding the IdM domain. - IPATrustCatalogCheck
This test resolves an AD user,
Administrator@REALM
. This populates the AD Global catalog and AD Domain Controller values insssctl domain-status
output.For each trust domain look up the user with the ID of the SID + 500, that is the administrator ID, and then check the output of
sssctl domain-status <domain> --active-server
to ensure that the domain is active.- IPAsidgenpluginCheck
-
This test verifies that the
sidgen
plugin is enabled in the IdM 389-ds instance. The test also verifies that theIPA SIDGEN
andipa-sidgen-task
plugins incn=plugins,cn=config
include thensslapd-pluginEnabled
option. - IPATrustAgentMemberCheck
-
This test verifies that the current host is a member of
cn=adtrust agents,cn=sysaccounts,cn=etc,SUFFIX
. - IPATrustControllerPrincipalCheck
-
This test verifies that the current host is a member of
cn=adtrust agents,cn=sysaccounts,cn=etc,SUFFIX
. - IPATrustControllerServiceCheck
- This test verifies that the current host starts the ADTRUST service in ipactl.
- IPATrustControllerConfCheck
-
This test verifies that
ldapi
is enabled for the passdb backend in the output ofnet conf
list. - IPATrustControllerGroupSIDCheck
-
This test verifies that the
admins
group’s SID ends with 512, which is the Domain Admins' RID. - IPATrustPackageCheck
-
This test verifies that the
trust-ad
package is installed if the trust controller and AD trust are not enabled.
8.2. Screening the trust with the Healthcheck tool
Follow this procedure to run a standalone manual test of an Identity Management (IdM) and Active Directory (AD) trust health check using the Healthcheck tool.
Procedure
Enter:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ipa-healthcheck --source=ipahealthcheck.ipa.trust --failures-only
# ipa-healthcheck --source=ipahealthcheck.ipa.trust --failures-only
-
The
--source=ipahealthcheck.ipa.trust
option ensures that IdM Healthcheck only performs the trust tests.
-
The
Successful test displays empty brackets:
ipa-healthcheck --source=ipahealthcheck.ipa.trust
# ipa-healthcheck --source=ipahealthcheck.ipa.trust
[]
Run these tests on all IdM servers when trying to find an issue.
Additional resources
Chapter 9. Verifying system certificates using IdM Healthcheck
Learn more about identifying issues with system certificates in Identity Management (IdM) by using the Healthcheck tool.
9.1. System certificates Healthcheck tests
The Healthcheck tool includes several tests for verifying system, or Dogtag, certificates.
You can find all certificate-related tests under the ipahealthcheck.dogtag.ca
source in the output of the ipa-healthcheck --list-sources
command.
- DogtagCertsConfigCheck
This test compares the CA (Certificate Authority) certificates in its NSS database to the same values stored in
CS.cfg
. If they do not match, the CA fails to start.Specifically, it checks:
-
auditSigningCert cert-pki-ca
againstca.audit_signing.cert
-
ocspSigningCert cert-pki-ca
againstca.ocsp_signing.cert
-
caSigningCert cert-pki-ca
againstca.signing.cert
-
subsystemCert cert-pki-ca
againstca.subsystem.cert
-
Server-Cert cert-pki-ca
againstca.sslserver.cert
If Key Recovery Authority (KRA) is installed, it also checks:
-
transportCert cert-pki-kra
againstca.connector.KRA.transportCert
-
- DogtagCertsConnectivityCheck
This test verifies connectivity. This test is equivalent to the
ipa cert-show 1
command which checks the following:- The PKI proxy configuration in Apache
- IdM being able to find a CA
- The RA agent client certificate
- The correctness of CA replies to requests
The test verifies that the
ipa cert-show
command can be executed and that an expected response is returned from the IdM CA - either the certificate itself or anot found
response.
9.2. Screening system certificates using Healthcheck
Follow this procedure to run a standalone manual test of Identity Management (IdM) certificates using the Healthcheck tool.
Procedure
Enter:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ipa-healthcheck --source=ipahealthcheck.dogtag.ca
# ipa-healthcheck --source=ipahealthcheck.dogtag.ca
-
The
--source=ipahealthcheck.dogtag.ca
option ensures that Healthcheck only performs the certificate tests.
-
The
An example of a successful test:
{ "source: ipahealthcheck.dogtag.ca", "check: DogtagCertsConfigCheck", "result: SUCCESS", "uuid: 9b366200-9ec8-4bd9-bb5e-9a280c803a9c", "when: 20191008135826Z", "duration: 0.252280", "kw:" { "key": "Server-Cert cert-pki-ca", "configfile": "/var/lib/pki/pki-tomcat/conf/ca/CS.cfg" } }
{
"source: ipahealthcheck.dogtag.ca",
"check: DogtagCertsConfigCheck",
"result: SUCCESS",
"uuid: 9b366200-9ec8-4bd9-bb5e-9a280c803a9c",
"when: 20191008135826Z",
"duration: 0.252280",
"kw:" {
"key": "Server-Cert cert-pki-ca",
"configfile": "/var/lib/pki/pki-tomcat/conf/ca/CS.cfg"
}
}
An example of a failed test:
{ "source: ipahealthcheck.dogtag.ca", "check: DogtagCertsConfigCheck", "result: CRITICAL", "uuid: 59d66200-1447-4b3b-be01-89810c803a98", "when: 20191008135912Z", "duration: 0.002022", "kw:" { "exception": "NSDB /etc/pki/pki-tomcat/alias not initialized", } }
{
"source: ipahealthcheck.dogtag.ca",
"check: DogtagCertsConfigCheck",
"result: CRITICAL",
"uuid: 59d66200-1447-4b3b-be01-89810c803a98",
"when: 20191008135912Z",
"duration: 0.002022",
"kw:" {
"exception": "NSDB /etc/pki/pki-tomcat/alias not initialized",
}
}
Run the certificate tests on all IdM servers when trying to find an issue.
Additional resources
Chapter 10. Verifying certificates using IdM Healthcheck
Learn more about understanding and using the Healthcheck tool in Identity management (IdM) to identify issues with IdM certificates maintained by the certmonger
utility.
10.1. IdM certificates Healthcheck tests
The Healthcheck tool includes several tests for verifying the status of certificates maintained by certmonger
in Identity Management (IdM). For details about certmonger, see Obtaining an IdM certificate for a service using certmonger.
This suite of tests checks certificate expiration, validation, trust, and other configuration. Healthcheck can report multiple errors for the same underlying issue.
You can find these certificate tests under the ipahealthcheck.ipa.certs
source in the output of the ipa-healthcheck --list-sources
command.
- IPACertmongerExpirationCheck
This test checks expirations in
certmonger
.If an error is reported, the certificate has expired.
If a warning appears, the certificate expires soon. By default, a warning appears if the test is run 28 days or fewer before certificate expiration.
You can configure the number of days in the
/etc/ipahealthcheck/ipahealthcheck.conf
file. After opening the file, change thecert_expiration_days
option located in thedefault
section.NoteCertmonger
loads and maintains its own view of the certificate expiration. This check does not validate the on-disk certificate.- IPACertfileExpirationCheck
This test checks if the certificate file or NSS database have correct access rights configured. This test also checks expiration. Therefore, carefully read the
msg
attribute in the error or warning output. The message specifies the problem.NoteThis test checks the on-disk certificate. If a certificate is missing or unreadable, Healthcheck returns an error.
- IPACertNSSTrust
- This test analyzes the trust for certificates stored in the NSS databases. For the expected tracked certificates in the NSS databases, Healthcheck compares the trust to an expected value and raises an error on a non-match.
- IPANSSChainValidation
-
This test validates the certificate chain of the NSS certificates. The test executes the
certutil -V -u V -e -d [dbdir] -n [nickname]
command. - IPAOpenSSLChainValidation
This test validates the certificate chain of the OpenSSL certificates. Specifically, Healthcheck executes the following OpenSSL command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow openssl verify -verbose -show_chain -CAfile /etc/ipa/ca.crt [cert file]
openssl verify -verbose -show_chain -CAfile /etc/ipa/ca.crt [cert file]
- IPARAAgent
-
This test compares the certificate on disk with the equivalent record in LDAP in
uid=ipara,ou=People,o=ipaca
. - IPACertRevocation
-
This test verifies that certificates that are maintained by
certmonger
have not been revoked. - IPACertmongerCA
This test verifies the
certmonger
Certificate Authority (CA) configuration. IdM cannot issue certificates without a CA.Certmonger
maintains a set of CA helpers. A CA namedIPA
issues certificates for hosts or services through IdM, authenticating as a host or user principal.There are also
dogtag-ipa-ca-renew-agent
anddogtag-ipa-ca-renew-agent-reuse
that renew the CA subsystem certificates.
10.2. Screening certificates using the Healthcheck tool
Follow this procedure to run a standalone manual test of an Identity Management (IdM) certificate health check using the Healthcheck tool.
Prerequisites
-
You have
root
privileges.
Procedure
Enter:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ipa-healthcheck --source=ipahealthcheck.ipa.certs --failures-only
# ipa-healthcheck --source=ipahealthcheck.ipa.certs --failures-only
The
--source=ipahealthcheck.ipa.certs
option ensures that IdM Healthcheck only performs thecertmonger
certificate tests.Successful test displays empty brackets:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow []
[]
Failed test shows you the following output:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow { "source": "ipahealthcheck.ipa.certs", "check": "IPACertfileExpirationCheck", "result": "ERROR", "kw": { "key": 1234, "dbdir": "/path/to/nssdb", "error": [error], "msg": "Unable to open NSS database '/path/to/nssdb': [error]" } }
{ "source": "ipahealthcheck.ipa.certs", "check": "IPACertfileExpirationCheck", "result": "ERROR", "kw": { "key": 1234, "dbdir": "/path/to/nssdb", "error": [error], "msg": "Unable to open NSS database '/path/to/nssdb': [error]" } }
This
IPACertfileExpirationCheck
test failed on opening the NSS database.
Run this suite of Healthcheck tests on all IdM servers when trying to check for issues.
Additional resources