Ignition troubleshooting

You can diagnose Ignition issues by examining logs, checking service status, and verifying configuration delivery.

Table 1. Ignition troubleshooting
Issue Cause Solution

Ignition does not run

Missing ignition.firstboot kernel parameter.

Add ignition.firstboot to the kernel command line in the boot loader configuration.

Configuration not found

Incorrect platform ID or missing configuration file.

Verify that ignition.platform.id matches your deployment method. Check the configuration location for your platform.

SSH keys not accepted

The sshd service is not configured to read the authorized_keys.d directory.

Add sshd configuration: AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys.d/ignition

Files not persisted

Writing to read-only file system locations.

Write files to the /var or /etc directory, which maps to /var on OSTree systems. Do not use /usr.

Ignition runs on every boot

First-boot marker not removed.

Verify ignition-firstboot-complete.service is enabled and runs successfully.

View Ignition logs

You can check the Ignition execution logs by using the journalctl utility.

  • To view all Ignition messages, run:
    # journalctl -t ignition --no-pager
  • To view logs for a specific Ignition service, such as ignition-disks.service, ignition-files.service, or ignition-fetch.service, run:
    # journalctl -u ignition-disks.service
  • To view the initramfs Ignition logs from the current boot, run:
    # journalctl -b 0 | grep -i ignition
Checking service status

You can verify the status of Ignition-related services by using the systemctl utility.

  • To check the status of the first-boot completion service, run:
    # systemctl status ignition-firstboot-complete.service
  • To check the status of the SSH key notification service, run:
    # systemctl status coreos-check-ssh-keys.service
  • To list all Ignition-related services, run:
    # systemctl list-units 'ignition' 'coreos'
Verify configuration delivery

You can check if Ignition successfully received its configuration depending on your deployment environment.

  • Inspect the Ignition configuration on bare metal

On bare-metal deployments, the Ignition configuration file is stored directly on the boot partition. To inspect the cached or copied configuration:

  1. Mount the boot partition:
    # mount /dev/disk/by-label/boot /mnt
  2. List the contents of the /mnt/ignition/ directory:
    # ls -la /mnt/ignition/
  3. View and format the contents of the Ignition configuration file:
    # cat /mnt/ignition/config.ign | jq .
  4. Unmount the boot partition:
    # umount /mnt
    • Verify Ignition configuration fetching under QEMU

When you pass an Ignition configuration to a QEMU instance by using initramfs, Ignition caches the downloaded payload in the /run directory.

  1. Check if Ignition cached the configuration file:
    # ls -la /run/ignition.json
  2. View the contents of the cached configuration file:
    # cat /run/ignition.json | jq .
    Debug in the initramfs

    If Ignition fails during boot, the system might drop to an emergency shell. You can use the following commands to diagnose the issue:

    • To check the status of the Ignition fetch service, run:
      # systemctl status ignition-fetch.service
    • To view the error output for the Ignition fetch service, run:
      # journalctl -u ignition-fetch.service
    • To check if the configuration source is accessible (QEMU fw_cfg example), run:
      # cat /sys/firmware/qemu_fw_cfg/by_name/opt/com.coreos/config/raw
    • To manually run Ignition in debug mode, run:
      # /usr/bin/ignition --platform=qemu --stage=fetch --log-to-stdout
    Resolve SSH key issues
    If SSH keys are not working on your deployed system, complete the following checks:
  3. Check where Ignition wrote the SSH keys:
    # ls -la /home/core/.ssh/
    # ls -la /home/core/.ssh/authorized_keys.d/
  4. Verify that the sshd service configuration is set up correctly:
    # grep -r AuthorizedKeysFile /etc/ssh/
  5. Check the sshd service logs for authorization events:
    # journalctl -u sshd.service | grep -i authorized
    Resolve first-boot marker issues
    If Ignition runs on subsequent boots, complete the following checks:
  6. Check if the first-boot marker exists:
    # ls -la /boot/ignition.firstboot
  7. Check the status of the first-boot completion service:
    # systemctl status ignition-firstboot-complete.service
    # journalctl -u ignition-firstboot-complete.service
  8. If necessary, manually remove the first-boot marker:
    # mount -o remount,rw /boot
    # rm -f /boot/ignition.firstboot
    # rm -rf /boot/ignition
    Resolve configuration validation errors

    You can validate your configuration files before deployment to catch syntax errors:

    • To validate an Ignition JSON file, run:
      # ignition-validate config.ign
    • To validate a Butane YAML file, run:
      $ butane --strict config.bu > /dev/null
    • To pretty-print and inspect the Ignition configuration, run:
      # cat config.ign | jq .
    Emergency console access
    If you cannot connect to the system by using SSH, use the following rescue steps:
  9. Connect to the system console (serial or VGA).
  10. Examine the console output for Ignition error messages during boot.
  11. If the system boots successfully, view the /etc/issue.d/ directory for status messages:
    $ cat /etc/issue.d/30_ssh_authorized_keys.issue
    Collect diagnostic information

    To gather system details for further troubleshooting, run the following diagnostic commands:

    • To view system information:
      # hostnamectl
      # rpm -qa | grep ignition
    • To view boot parameters:
      # cat /proc/cmdline
    • To save the current boot’s Ignition logs to a file:
      # journalctl -b 0 -t ignition > ignition.log
    • To save a list of failed services to a file:
      # systemctl list-units --failed > failed-units.log