Using IdM Healthcheck to monitor your IdM environment

The Healthcheck tool in Identity Management (IdM) helps find issues that may impact the health of your IdM environment.

You can specify a different file destination with the --output-file option.

Follow this procedure to install the IdM Healthcheck tool.

Healthcheck can be run manually or automatically using log rotation

Log rotation creates a new log file every day, and the files are organized by date. Since log files are saved in the same directory, you can select a particular log file according to the date.

Rotation means that there is configured a number for max number of log files and if the number is exceeded, the newest file rewrites and renames the oldest one. For example, if the rotation number is 30, the thirty-first log file replaces the first (oldest) one.

Log rotation reduces voluminous log files and organizes them, which can help with analysis of the logs.

Follow this procedure to configure a log rotation with:

The systemd timer runs the Healthcheck tool periodically and generates the logs. The default value is set to 4 a.m. every day.

The crond service is used for log rotation.

The default log name is healthcheck.log and the rotated logs use the healthcheck.log-YYYYMMDD format.

To start with generating logs, start the IPA healthcheck service:

# systemctl start ipa-healthcheck

To verify the result, go to /var/log/ipa/healthcheck/ and check if logs are created correctly.

You can change Healthcheck settings by adding the desired command line options to the /etc/ipahealthcheck/ipahealthcheck.conf file. This can be useful when, for example, you configured a log rotation and want to ensure the logs are in a format suitable for automatic analysis, but do not want to set up a new timer.

After the modification, all logs that Healthcheck creates follow the new settings. These settings also apply to any manual execution of Healthcheck.

Follow this procedure to configure Healthcheck with a timer already set up. In this example, you configure Healthcheck to produce logs in a human-readable format and to also include successful results instead of only errors.

The Healthcheck tool includes a test to check if any IdM services is not running. This test is important because services which are not running can cause failures in other tests. Therefore, check that all services are running first. You can then check all other test results.

To see all services tests, run ipa-healthcheck with the --list-sources option:

# ipa-healthcheck --list-sources

You can find all services tested with Healthcheck under the ipahealthcheck.meta.services source:

Follow this procedure to run a standalone manual test of services running on the Identity Management (IdM) server using the Healthcheck tool.

The Healthcheck tool includes many tests, whose results can be shortened with:

A successful test displays empty brackets:

[ ]

If one of the services fails, the result can looks similarly to this example:

{
  "source": "ipahealthcheck.meta.services",
  "check": "httpd",
  "result": "ERROR",
  "kw": {
    "status": false,
    "msg": "httpd: not running"
  }
}

The Healthcheck tool includes a test for checking available disk space. Insufficient free disk space can cause issues with:

The test checks the following paths:

To list all tests, run the ipa-healthcheck with the --list-sources option:

# ipa-healthcheck --list-sources

You can find the file system space check test under the ipahealthcheck.system.filesystemspace source:

Follow this procedure to run a standalone manual test of available disk space on an Identity Management (IdM) server using the Healthcheck tool.

Since Healthcheck includes many tests, you can narrow the results by:

A successful test displays empty brackets:

[]

As an example, a failed test can display:

{
  "source": "ipahealthcheck.system.filesystemspace",
  "check": "FileSystemSpaceCheck",
  "result": "ERROR",
  "kw": {
    "msg": "/var/lib/dirsrv: free space under threshold: 0 MiB < 1024 MiB",
    "store": "/var/lib/dirsrv",
    "free_space": 0,
    "threshold": 1024
  }
}

The failed test informs you that the /var/lib/dirsrv directory has run out of space.

The Healthcheck tool tests ownership and permissions of some important files installed or configured by Identity Management (IdM).

If you change the ownership or permissions of any tested file, the test returns a warning in the result section. While it does not necessarily mean that the configuration will not work, it means that the file differs from the default configuration.

To see all tests, run the ipa-healthcheck with the --list-sources option:

# ipa-healthcheck --list-sources

You can find the file permissions test under the ipahealthcheck.ipa.files source:

Follow this procedure to run a standalone manual test of an Identity Management (IdM) server’s configuration files using the Healthcheck tool.

The Healthcheck tool includes many tests. Results can be narrowed down by:

A successful test displays empty brackets:

# ipa-healthcheck --source=ipahealthcheck.ipa.files --failures-only
[]

Failed tests display results similar to the following WARNING:

{
  "source": "ipahealthcheck.ipa.files",
  "check": "IPAFileNSSDBCheck",
  "result": "WARNING",
  "kw": {
    "key": "_etc_dirsrv_slapd-EXAMPLE-TEST_pkcs11.txt_mode",
    "path": "/etc/dirsrv/slapd-EXAMPLE-TEST/pkcs11.txt",
    "type": "mode",
    "expected": "0640",
    "got": "0666",
    "msg": "Permissions of /etc/dirsrv/slapd-EXAMPLE-TEST/pkcs11.txt are 0666 and should be 0640"
  }
}

The Healthcheck tool includes a test for checking that the expected DNS records required for autodiscovery are resolvable.

To list all tests, run the ipa-healthcheck with the --list-sources option:

# ipa-healthcheck --list-sources

You can find the DNS records check test under the ipahealthcheck.ipa.idns source.

Follow this procedure to run a standalone manual test of DNS records on an Identity Management (IdM) server using the Healthcheck tool.

The Healthcheck tool includes many tests. Results can be narrowed down by including only the DNS records tests by adding the --source ipahealthcheck.ipa.idns option.

The Healthcheck tool tests the Identity Management (IdM) topology configuration and searches for replication conflict issues.

To list all tests, run the ipa-healthcheck with the --list-sources option:

# ipa-healthcheck --list-sources

The topology tests are placed under the ipahealthcheck.ipa.topology and ipahealthcheck.ds.replication sources:

Follow this procedure to run a standalone manual test of an Identity Management (IdM) replication topology and configuration using the Healthcheck tool.

The Healthcheck tool includes many tests. Therefore, you can shorten the results with:

The Healthcheck tool includes several tests for testing the status of your Identity Management (IdM) and Active Directory (AD) trust.

To see all trust tests, run ipa-healthcheck with the --list-sources option:

# ipa-healthcheck --list-sources

You can find all tests under the ipahealthcheck.ipa.trust source:

Follow this procedure to run a standalone manual test of an Identity Management (IdM) and Active Directory (AD) trust health check using the Healthcheck tool.

The Healthcheck tool includes many tests, therefore, you can shorten the results by:

Successful test displays empty brackets:

# ipa-healthcheck --source=ipahealthcheck.ipa.trust --failures-only
[]

The Healthcheck tool includes several tests for verifying system (DogTag) certificates.

To see all tests, run the ipa-healthcheck with the --list-sources option:

# ipa-healthcheck --list-sources

You can find all tests under the ipahealthcheck.dogtag.ca source:

Follow this procedure to run a standalone manual test of Identity Management (IdM) certificates using the Healthcheck tool.

Since, the Healthcheck tool includes many tests, you can narrow the results by including only DogTag tests: --source=ipahealthcheck.dogtag.ca

An example of a successful test:

{
  "source: ipahealthcheck.dogtag.ca",
  "check: DogtagCertsConfigCheck",
  "result: SUCCESS",
  "uuid: 9b366200-9ec8-4bd9-bb5e-9a280c803a9c",
  "when: 20191008135826Z",
  "duration: 0.252280",
  "kw:" {
    "key": "Server-Cert cert-pki-ca",
    "configfile":  "/var/lib/pki/pki-tomcat/conf/ca/CS.cfg"
    }
}

An example of a failed test:

{
  "source: ipahealthcheck.dogtag.ca",
  "check: DogtagCertsConfigCheck",
  "result: CRITICAL",
  "uuid: 59d66200-1447-4b3b-be01-89810c803a98",
  "when: 20191008135912Z",
  "duration: 0.002022",
  "kw:" {
    "exception": "NSDB /etc/pki/pki-tomcat/alias not initialized",
    }
}

The Healthcheck tool includes several tests for verifying the status of certificates maintained by certmonger in Identity Management (IdM). For details about certmonger, see Obtaining an IdM certificate for a service using certmonger.

This suite of tests checks expiration, validation, trust and other issues. Multiple errors may be thrown for the same underlying issue.

To see all certificate tests, run the ipa-healthcheck with the --list-sources option:

# ipa-healthcheck --list-sources

You can find all tests under the ipahealthcheck.ipa.certs source:

Follow this procedure to run a standalone manual test of an Identity Management (IdM) certificate health check using the Healthcheck tool.

The Healthcheck tool includes many tests, therefore, you can shorten the results with:

Successful test displays empty brackets:

[]

Failed test shows you the following output:

{
  "source": "ipahealthcheck.ipa.certs",
  "check": "IPACertfileExpirationCheck",
  "result": "ERROR",
  "kw": {
    "key": 1234,
    "dbdir": "/path/to/nssdb",
    "error": [error],
    "msg": "Unable to open NSS database '/path/to/nssdb': [error]"
  }
}

This IPACertfileExpirationCheck test failed on opening the NSS database.