Ignition troubleshooting
You can diagnose Ignition issues by examining logs, checking service status, and verifying configuration delivery.
| Issue | Cause | Solution |
|---|---|---|
|
Ignition does not run |
Missing |
Add |
|
Configuration not found |
Incorrect platform ID or missing configuration file. |
Verify that |
|
SSH keys not accepted |
The |
Add |
|
Files not persisted |
Writing to read-only file system locations. |
Write files to the |
|
Ignition runs on every boot |
First-boot marker not removed. |
Verify |
- View Ignition logs
-
You can check the Ignition execution logs by using the
journalctlutility.- To view all Ignition messages, run:
# journalctl -t ignition --no-pager - To view logs for a specific Ignition service, such as
ignition-disks.service,ignition-files.service, orignition-fetch.service, run:# journalctl -u ignition-disks.service - To view the
initramfsIgnition logs from the current boot, run:# journalctl -b 0 | grep -i ignition
- To view all Ignition messages, run:
- Checking service status
-
You can verify the status of Ignition-related services by using the
systemctlutility.- To check the status of the first-boot completion service, run:
# systemctl status ignition-firstboot-complete.service - To check the status of the SSH key notification service, run:
# systemctl status coreos-check-ssh-keys.service - To list all Ignition-related services, run:
# systemctl list-units 'ignition' 'coreos'
- To check the status of the first-boot completion service, run:
- Verify configuration delivery
-
You can check if Ignition successfully received its configuration depending on your deployment environment.
- Inspect the Ignition configuration on bare metal
On bare-metal deployments, the Ignition configuration file is stored directly on the boot partition. To inspect the cached or copied configuration:
- Mount the boot partition:
# mount /dev/disk/by-label/boot /mnt - List the contents of the
/mnt/ignition/directory:# ls -la /mnt/ignition/ - View and format the contents of the Ignition configuration file:
# cat /mnt/ignition/config.ign | jq . - Unmount the boot partition:
# umount /mnt- Verify Ignition configuration fetching under QEMU
When you pass an Ignition configuration to a QEMU instance by using initramfs, Ignition caches the downloaded payload in the /run directory.
- Check if Ignition cached the configuration file:
# ls -la /run/ignition.json - View the contents of the cached configuration file:
# cat /run/ignition.json | jq .- Debug in the initramfs
If Ignition fails during boot, the system might drop to an emergency shell. You can use the following commands to diagnose the issue:
- To check the status of the Ignition fetch service, run:
# systemctl status ignition-fetch.service - To view the error output for the Ignition fetch service, run:
# journalctl -u ignition-fetch.service - To check if the configuration source is accessible (QEMU
fw_cfgexample), run:# cat /sys/firmware/qemu_fw_cfg/by_name/opt/com.coreos/config/raw - To manually run Ignition in debug mode, run:
# /usr/bin/ignition --platform=qemu --stage=fetch --log-to-stdout
- To check the status of the Ignition fetch service, run:
- Resolve SSH key issues
- If SSH keys are not working on your deployed system, complete the following checks:
- Check where Ignition wrote the SSH keys:
# ls -la /home/core/.ssh/# ls -la /home/core/.ssh/authorized_keys.d/ - Verify that the
sshdservice configuration is set up correctly:# grep -r AuthorizedKeysFile /etc/ssh/ - Check the
sshdservice logs for authorization events:# journalctl -u sshd.service | grep -i authorized- Resolve first-boot marker issues
- If Ignition runs on subsequent boots, complete the following checks:
- Check if the first-boot marker exists:
# ls -la /boot/ignition.firstboot - Check the status of the first-boot completion service:
# systemctl status ignition-firstboot-complete.service# journalctl -u ignition-firstboot-complete.service - If necessary, manually remove the first-boot marker:
# mount -o remount,rw /boot# rm -f /boot/ignition.firstboot# rm -rf /boot/ignition- Resolve configuration validation errors
You can validate your configuration files before deployment to catch syntax errors:
- To validate an Ignition JSON file, run:
# ignition-validate config.ign - To validate a Butane YAML file, run:
$ butane --strict config.bu > /dev/null - To pretty-print and inspect the Ignition configuration, run:
# cat config.ign | jq .
- To validate an Ignition JSON file, run:
- Emergency console access
- If you cannot connect to the system by using SSH, use the following rescue steps:
- Connect to the system console (serial or VGA).
- Examine the console output for Ignition error messages during boot.
- If the system boots successfully, view the
/etc/issue.d/directory for status messages:$ cat /etc/issue.d/30_ssh_authorized_keys.issue- Collect diagnostic information
To gather system details for further troubleshooting, run the following diagnostic commands:
- To view system information:
# hostnamectl# rpm -qa | grep ignition - To view boot parameters:
# cat /proc/cmdline - To save the current boot’s Ignition logs to a file:
# journalctl -b 0 -t ignition > ignition.log - To save a list of failed services to a file:
# systemctl list-units --failed > failed-units.log
- To view system information: