Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Managing device fleets with the Red Hat Edge Manager

Red Hat Ansible Automation Platform 2.6

Install, configure, and use the Red Hat Edge Manager to manage individual and fleets of devices

Red Hat Customer Content Services

Abstract

Learn about components that you can use for scalable and secure edge management.

Preface
Copy link

The Red Hat Edge Manager aims to give simple, scalable, and secure management of edge devices and applications. You can declare the operating system version, host configuration, and set of applications that you want to run on an individual device or a whole fleet of devices. The Red Hat Edge Manager rolls out the target configuration to devices where a device agent automatically applies them and reports progress and health status back up.

Important

The Red Hat Edge Manager is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Providing feedback on Red Hat documentation
Copy link

If you have a suggestion to improve this documentation, or find an error, you can contact technical support at https://access.redhat.com to open a request.

Chapter 1. Red Hat Edge Manager overview
Copy link

The Red Hat Edge Manager provides streamlined management of edge devices and applications through a declarative approach. By defining the required state of your edge devices, which includes your operating system versions, host configurations, and application deployments, the Red Hat Edge Manager automatically implements and maintains these configurations across your entire device fleet.

The Red Hat Edge Manager on Ansible Automation Platform offers elevated integration with your automations. You can then focus more on orchestrating the environment without worrying about updating the operating system.

See the following topics to learn more about using Red Hat Edge Manager on Ansible Automation Platform:

Chapter 2. Red Hat Edge Manager architecture
Copy link

You can manage individual devices or an entire fleet by using the Red Hat Edge Manager. The Red Hat Edge Manager uses an agent-based architecture that allows for a scalable and robust device management, even with limited network conditions.

By deploying a Red Hat Edge Manager agent to a device, the agent autonomously manages and monitors the device while periodically communicating with the Red Hat Edge Manager service to check for new configurations and to report device status.

The Red Hat Edge Manager supports image-based operating systems. You can include the Red Hat Edge Manager agent and the agent configuration in the image that is distributed to the devices.

Image-based operating systems allow the agent to start a transactional update of the image and to roll back to the earlier version in case of an update error.

The Red Hat Edge Manager architecture has the following main features:

  • Agent
  • Service
  • Image-based operating system
  • API server
  • Database
  • Device
  • Device fleet

Learn more from the following sections:

2.1. Red Hat Edge Manager agent and service
Copy link

The Red Hat Edge Manager agent is a process running on each managed device that periodically communicates with the Red Hat Edge Manager service. The agent is responsible for the following tasks:

  • Enrolling devices into the service
  • Periodically checking with the service for changes in the device specification, such as changes to the operating system, configuration, and applications
  • Applying any updates independently from the service
  • Reporting status of the device and the applications

The Red Hat Edge Manager service is responsible for the following tasks:

  • Authenticating and authorizing users and agents
  • Enrolling devices
  • Managing device inventory
  • Reporting status from individual devices or fleets

The service also communicates with a database that stores the device inventory and the target device configuration. When communicating with the service, the agent polls the service for changes in the configuration. If the agent detects that the current configuration deviates from the target configuration, the agent attempts to apply the changes to the device.

When the agent receives a new target configuration from the service, the agent does the following tasks:

  1. To avoid depending on network connectivity during the update, the agent downloads all required resources, such as the operating system image and application container images, over the network to disk.
  2. The agent updates the operating system image by delegating to bootc.
  3. The agent updates configuration files on the file system of the device by overlaying a set of files that the service sends to the device.
  4. If necessary, the agent reboots into the new operating system. Otherwise, the agent signals system services and applications to reload the updated configuration.
  5. The agent updates applications running on Podman.

If the update fails or the system does not return online after rebooting, the agent automatically rolls back to the earlier operating system image and configuration.

Note

You can keep fleet definitions in Git. The Red Hat Edge Manager periodically syncs with the fleet definitions in the database.

2.2. Red Hat Edge Manager API server
Copy link

The API server is a core part of the Red Hat Edge Manager service that gives users and agents an option to communicate with the service.

The API server exposes the following endpoints:

User-facing API endpoint
Users can connect to the user-facing API endpoint from the CLI or the web console. Users must authenticate on the platform gateway to obtain a JSON Web Token (JWT) to make HTTPS requests.
Agent-facing API endpoint
Agents connect to the agent-facing endpoint, which is mTLS-protected. The service authenticates devices by using the X.509 client certificates.

The Red Hat Edge Manager service also communicates with various external systems to authenticate and authorize users, get mTLS certificates signed, or query configuration for managed devices.

2.3. Device enrollment
Copy link

You must enroll devices to a Red Hat Edge Manager service before you can start managing them. The Red Hat Edge Manager agent that runs on a device handles the device enrollment.

When the agent starts on a device, the agent searches for the configuration in the /etc/flightctl/config.yaml file. The file defines the following configurations:

  • The enrollment endpoint, which is the Red Hat Edge Manager service that the agent connects to for enrollment.
  • The enrollment certificate, which is the X.509 client certificate and key that the agent only uses to securely request enrollment from the Red Hat Edge Manager service.
  • Optional: Any additional agent configuration.

The agent starts the enrollment process by searching for the enrollment endpoint, the Red Hat Edge Manager service, that is defined in the configuration file. After establishing a secure, mTLS-protected network connection with the service, the agent submits an enrollment request to the service.

The request includes a description of hardware and operating system of the device, a X.509 certificate signing request, and the cryptographic identity of the device. The enrollment request must be approved by an authorized user. After the request is approved, the device becomes trusted and managed by the Red Hat Edge Manager service.

2.3.1. Enrollment methods
Copy link

You can provision the enrollment endpoint and certificate to the device in the following ways:

Early binding
You can build an operating system image that includes the enrollment endpoint and certificate. Devices using an early binding image can automatically connect to the defined service to request enrollment, without depending on any provisioning infrastructure. The devices share the same long-lived X.509 client certificate. However, in this case, the devices are bound to a specific service and owner.
Late binding
You can define the enrollment endpoint and certificate at provisioning time instead of including them in the operating system image. Devices using a late binding image are not bound to a single owner or service and can have device-specific, short-lived X.509 client certificates. However, late binding requires virtualization or bare-metal provisioning infrastructure that can request device-specific enrollment endpoints and certificates from the Red Hat Edge Manager service and inject them into the provisioned system by using mechanisms such as cloud-init, Ignition, or kickstart.
Note

The enrollment certificate is only used to secure the network connection for submitting an enrollment request. The enrollment certificate is not involved in the actual verification or approval of the enrollment request. The enrollment certificate is no longer used with enrolled devices, as the devices rely on device-specific management certificates instead.

Install the Red Hat Edge Manager to manage edge devices and applications at scale. This guide focuses on a standalone deployment of the Red Hat Edge Manager on Red Hat Enterprise Linux alongside Ansible Automation Platform.

Prepare your Red Hat Enterprise Linux host for the installation of the Red Hat Edge Manager by enabling the necessary repositories, installing the flightctl-services package, configuring the baseDomain, and then starting and verifying the running services.

Prerequisites

  • An active Ansible Automation Platform subscription with a running instance and the necessary API URLs and OAuth credentials.
  • A separate machine from Ansible Automation Platform to install the Red Hat Edge Manager on.
  • Podman installed for managing containers.

  • A Red Hat Enterprise Linux host with:

    • Minimal installation
    • 4 cores and 16GB RAM (recommended)
    • Administrative access (root or sudo-capable user)
    • SSH access

Procedure

  1. SSH into your Red Hat Enterprise Linux host.

  2. Authenticate and log in to the Red Hat Container Registry: 

    sudo podman login registry.redhat.io
    Copy to Clipboard Toggle word wrap

  3. Install the necessary repositories and packages:

    • Ensure that the Ansible Automation Platform repositories are enabled by running the following example command based on the version of Red Hat Enterprise Linux and architecture of your host: 

      sudo subscription-manager repos --enable ansible-automation-platform-2.5-for-rhel-9-x86_64-rpms
      Copy to Clipboard Toggle word wrap

    • Install the Red Hat Edge Manager service by running: 

      sudo dnf install -y flightctl-services
      Copy to Clipboard Toggle word wrap

  4. Update the installed /etc/flightctl/service-config.yaml to set the baseDomain: 

    sudo vi /etc/flightctl/service-config.yaml
    Copy to Clipboard Toggle word wrap
    Important

    Ensure that you set the baseDomain in the service configuration correctly. By default, the installation process attempts to automatically set this value based on the IP address of your Red Hat Enterprise Linux host.

    However, if your environment uses a specific domain name to access this host, for example rhem-example.com, it is recommended that you manually update the baseDomain in /etc/flightctl/service-config.yaml to this hostname.

    Setting the baseDomain correctly ensures that all generated URLs, certificates, and internal configurations within the Red Hat Edge Manager are accurate for your network setup. This is especially important for integration with Ansible Automation Platform and for ensuring that the UI is accessible through the intended domain name.

    You can check the currently configured baseDomain using: 

    grep baseDomain: /etc/flightctl/service-config.yaml
    Copy to Clipboard Toggle word wrap

  5. Enable and start the services: 

    sudo systemctl enable flightctl.target
sudo systemctl start flightctl.target
    Copy to Clipboard Toggle word wrap

  6. Verify that services are running: 

    sudo systemctl list-units flightctl-*.service
    Copy to Clipboard Toggle word wrap

    You should see these 7 services running:

    • flightctl-db
    • flightctl-kv
    • flightctl-api
    • flightctl-periodic
    • flightctl-worker
    • flightctl-ui
    • flightctl-cli-artifacts

  7. Go to the UI at the baseDomain stored in the service configuration file: 

    grep baseDomain: /etc/flightctl/service-config.yaml
    Copy to Clipboard Toggle word wrap

    Visit the displayed baseDomain in your web browser to access the UI.

Troubleshooting

If your services do not run correctly, use the following log command to troubleshoot further and remediate: 

journalctl -u flightctl-<impacted service> -b --no-pager
Copy to Clipboard Toggle word wrap

You have two options for setting up the OAuth application in Ansible Automation Platform, either manually or automatically in the Ansible Automation Platform UI.

Automatic setup of an OAuth application by generating an OAuth token within Ansible Automation Platform and adding it to your configuration file. Upon service startup, the application is automatically created, and the client ID updated.

Procedure

  1. Generate an OAuth token in Ansible Automation Platform:

    1. From the navigation panel, select Access Management → Users.
    2. Select a user with write permissions to the Default organization (admin user recommended).
    3. Click the Tokens tab for that user.

    4. Click Create token and enter the relevant details.

      1. Scope: Select Write.
  2. Go to the Integrating with Ansible Automation Platform section for the steps to edit your service-config.yaml file and complete setting up the OAuth application automatically.

3.2.2. Setting up the OAuth application manually
Copy link

Manually set up an OAuth application within your Ansible Automation Platform instance. This is important for enabling token-based authentication and integrating external applications such as the Red Hat Edge Manager.

Procedure

  1. From the navigation panel on your Ansible Automation Platform instance, go to Access Management → OAuth Applications.
  2. Click Create OAuth application.

  3. Enter the following details:

    • Name: Enter a name such as "Red Hat Edge Manager". This is the name visible in the Ansible Automation Platform UI.
    • URL: The baseDomain of your Red Hat Edge Manager UI with https://.
    • Organization: Select Default.
    • Authorization grant type: Select Authorization code.
    • Client: Select Public.

    • Redirect URIs:

  4. Click Create OAuth application. An Application Links section is now visible in the navigation panel.
  5. Copy the Client ID as you need it to update the oAuthApplicationClientId in your service-config.yaml file with this value.
  6. Go to the Integrating with Ansible Automation Platform section for the steps to edit your service-config.yaml file and complete setting up the OAuth application manually.

Additional resources

Integrate the Red Hat Edge Manager with your Ansible Automation Platform instance by modifying the service-config.yaml file to include authentication type, API URLs, OAuth client ID, and an optional OAuth token, followed by restarting the services.

Procedure

  1. Stop the flightctl services before editing your service-config.yaml file: 

    sudo systemctl stop flightctl.target
    Copy to Clipboard Toggle word wrap

  2. Configure the integration settings by editing the configuration file: 

    sudo vi /etc/flightctl/service-config.yaml
    Copy to Clipboard Toggle word wrap

  3. Update the configuration file to integrate with Ansible Automation Platform: 

    global:
  baseDomain: <your-edge-manager-ip-or-domain>
    1 
    
  auth:
    type: aap
    2 
    
    insecureSkipTlsVerify: false
    3 
    
    aap:
      apiUrl: https://your-aap-instance.example.com
    4 
    
      externalApiUrl: https://your-aap-instance.example.com
    5 
    
      oAuthApplicationClientId: <client-id-from-oauth-app>
    6 
    
      oAuthToken: <your-oauth-token>
    7
    Copy to Clipboard Toggle word wrap
    1
    The domain name or IP for the host, this is automatically set when the RPM is installed but you can override this. It is the only field that is mandatory.
    2
    Set this to aap to enable Ansible Automation Platform authentication.
    3
    Set to false. Only set this to true to skip TLS certificate verification for the Ansible Automation Platform URLs. For production environments, consider configuring a CA certificate (see the Self-signed certificates section).
    4
    The internal facing API URL for the running Ansible Automation Platform instance that makes requests against. You can configure this URL to be an internally accessible URL for the running Ansible Automation Platform instance. For example, if there are separate internal or external ingresses.
    5
    The externally accessible URL of your running Ansible Automation Platform instance.
    6
    If you are using the automatic method, this field is not necessary. This is the Client ID of the OAuth application configured in Ansible Automation Platform for the Red Hat Edge Manager. If you do not have one yet, you can leave this empty and give an oAuthToken to allow the setup to create it.
    7
    If you are using the manual method, this field is not necessary. This is an OAuth token with write permissions for the "Default" organization in your Ansible Automation Platform instance. This is only needed if you want the setup process to automatically create the OAuth application. Once created, this token is no longer necessary.

  4. Start the services: 

    sudo systemctl start flightctl.target
    Copy to Clipboard Toggle word wrap

3.3. Self-signed certificates
Copy link

The Red Hat Edge Manager services automatically generate and store self-signed certificates in the /etc/flightctl/pki directory. These include:

  • /etc/flightctl/pki/ca.crt
  • /etc/flightctl/pki/ca.key
  • /etc/flightctl/pki/client-enrollment.crt
  • /etc/flightctl/pki/client-enrollment.key
  • /etc/flightctl/pki/server.crt
  • /etc/flightctl/pki/server.key

You can use your own custom certificates by placing them in the following locations:

  • Custom Server Certificate/Key Pair:

    • /etc/flightctl/pki/server.crt
    • /etc/flightctl/pki/server.key

  • Custom CA Certificate for Ansible Automation Platform authentication:

    • /etc/flightctl/pki/auth/ca.crt
Note

Ensure that you adjust the insecureSkipTlsVerify setting in the service-config.yaml if you use a custom CA certificate for your Ansible Automation Platform instance.

Image-based operating systems allow the operating system and its configuration and applications to be versioned, deployed, and updated as a single unit. Using an image-based operating system reduces operational risks by doing the following:

  • Minimizing potential drift between what is tested and what is deployed to a large number of devices.
  • Minimizing the risk of failed updates that require expensive maintenance or replacement through transactional updates and rollbacks.

The Red Hat Edge Manager focuses on image-based Linux operating systems that run bootable container images (bootc).

For more information, see bootc.

Important

The bootc tool does not update package-based operating systems.

4.1. The image building process
Copy link

  1. Choose a base bootc operating system image, such as a Fedora, CentOS, or RHEL image.

  2. Create a container file that layers the following items onto the base bootc image:

    • The Red Hat Edge Manager agent and configuration.
    • Optional: Any drivers specific to your target deployment environment.
    • Optional: Host configuration, for example, certificate authority bundles, and application workloads that are common to all deployments.
  3. Build, publish, and sign a bootc operating system image using podman and skopeo.
  4. Create an operating system disk image by using bootc-image-builder.
  5. Build, publish, and sign an operating system disk image using skopeo.
Note

The operating system disk image has partitions, volumes, the file system, and the initial bootc image. You only need to create the operating system disk image once, during provisioning. For later device updates, you only need the bootc operating system image, which has the files in the file system.

Additional resources

4.2. Special considerations for building images
Copy link

Add the configuration to the operating system image at build time. Adding the configuration at build time ensures that the configurations are tested, distributed, and updated together. In cases when build-time configuration is not feasible or desirable, you can dynamically configure devices at runtime instead with the Red Hat Edge Manager.

Dynamic runtime configuration is preferable in the following cases:

  • You have a configuration that is deployment or site-specific, such as a hostname or a site-specific network credential.
  • You have secrets that are not secure to distribute with the image.
  • You have application workloads that need to be added, updated, or deleted without reboot or they are on a faster cadence than the operating system.

4.2.2. Configuration in the /usr directory
Copy link

Place configuration files in the /usr directory if the configuration is static and the application or service supports that configuration. By placing the configuration in the /usr directory, the configuration remains read-only and fully defined by the image.

Do not place the configuration in the /usr directory in the following cases:

  • The configuration is deployment or site-specific.
  • The application or service only supports reading configuration from the /etc directory.
  • The configuration might need to be changed at runtime.

4.2.3. Drop-in directories
Copy link

Use drop-in directories to add, replace, or remove configuration files that the service aggregates. Do not directly edit your configuration files because it can cause deviations from the target configuration.

Note

You can identify drop-in directories by the .d/ at the end of the directory name. For example, /etc/containers/certs.d, /etc/cron.d, and /etc/NetworkManager/conf.d.

4.2.4. Operating system images with scripts
Copy link

Avoid executing scripts or commands that change the file system. The bootc or the Red Hat Edge Manager can overwrite the changed files which can cause a deviation or failed integrity checks..

Instead, run such scripts or commands during image building so changes are part of the image. You can also use the configuration management mechanisms of the Red Hat Edge Manager.

Additional resources

To prepare your device to be managed by the Red Hat Edge Manager, build a bootc operating system image that has the Red Hat Edge Manager agent. Then build an operating system disk image for your devices.

For more information, see the following sections:

4.3.1. Prerequisites
Copy link

See the following prerequisites for building a bootc operating system image:

4.3.2. Installing the Red Hat Edge Manager CLI
Copy link

To install the Red Hat Edge Manager CLI, complete the following steps:

Procedure

  1. Enable the subscription manager for the repository appropriate for your system by running the following command: 

    sudo subscription-manager repos --enable ansible-automation-platform-2.5-for-rhel-9-x86_64-rpms
    Copy to Clipboard Toggle word wrap

    For a full list of available repositories for the Red Hat Edge Manager, see the Additional resources section.

  2. Install the flightctl CLI with your package manager by running the following command: 

    sudo dnf install flightctl
    Copy to Clipboard Toggle word wrap

If you set up the OAuth application manually, you also need to make sure that one utility xdg-open, x-www-browser, or www-browser is available, for example, by installing xdg-utils.

How you log in the Red Hat Edge Manager depends on whether you choose the automatic or manual method when you initially set up the application.

Procedure

  • If you use the automatic setup you can create a personal access token, even only with Read scope (under the profile icon in the top right corner of your Ansible Automation Platform UI > User details > Tokens tab) and then use this token to log in directly through the CLI, with the following example syntax: 

    flightctl login https://<your-edge-manager-ip-or-domain>:3443 --token=<your-aap-oauth-token> --insecure-skip-tls-verify
    Copy to Clipboard Toggle word wrap

  • If you use the manual setup, use the Client ID to log in through a web-based process, with the following example syntax: 

    flightctl login https://<your-edge-manager-ip-or-domain>:3443 --web --client-id=<your-aap-client-id> --insecure-skip-tls-verify
    Copy to Clipboard Toggle word wrap

    • This opens in a web browser and asks you to approve.

      The --insecure-skip-tls-verify parameter is used only if you have not generated your own valid certificates.

Next steps

Use the following commands to help you with the CLI:

  • To output a list of available commands, use: 

    flightctl
    Copy to Clipboard Toggle word wrap

  • To output both the flightctl CLI version and the back-end Red Hat Edge Manager version, use: 

    flightctl version
    Copy to Clipboard Toggle word wrap
Important

To ensure supportability and proper functionality, the version of the flightctl CLI must match the version of the Red Hat Edge Manager in use. Mismatched versions are not supported.

If you want to include an agent configuration in the image, complete the following steps:

Procedure

  1. Log in to the flightctl CLI by following the steps in Logging into the Red Hat Edge Manager through the CLI.

    Note

    The CLI uses the certificate authority pool of the host to verify the identity of the Red Hat Edge Manager service. The verification can lead to a TLS verification error when using self-signed certificates, if you do not add your certificate authority certificate to the pool. You can bypass the server verification by adding the --insecure-skip-tls-verify flag to your command.

  2. Get the enrollment credentials in the format of an agent configuration file by running the following command: 

    flightctl certificate request --signer=enrollment --expiration=365d --output=embedded > config.yaml
    Copy to Clipboard Toggle word wrap
    Note
    • The --expiration=365d option specifies that the credentials are valid for a year.
    • The --output=embedded option specifies that the output is an agent configuration file with the enrollment credentials embedded.

    The returned config.yaml contains the URLs of the Red Hat Edge Manager service, the certificate authority bundle, and the enrollment client certificate and key for the agent. See the following example: 

    enrollment-service:
  authentication:
    client-certificate-data: LS0tLS1CRUdJTiBD...
    client-key-data: LS0tLS1CRUdJTiBF...
  service:
    certificate-authority-data: LS0tLS1CRUdJTiBD...
    server: https://agent-api.flightctl.127.0.0.1.nip.io:7443
  enrollment-ui-endpoint: https://ui.flightctl.127.0.0.1.nip.io:8081
    Copy to Clipboard Toggle word wrap

4.3.5. Optional: Using image pull secrets
Copy link

If your device relies on containers from a private repository, you must configure a pull secret for the registry. Complete the following steps:

Procedure

  1. Depending on the kind of container image you use, place the pull secret in one or both of the following system paths on the device:

    • Operating system images use the /etc/ostree/auth.json path.

    • Application container images use the /root/.config/containers/auth.json path.

      Important

      The pull secret must exist on the device before the secret can be consumed.

  2. Ensure that the pull secrets use the following format: 

    {
  "auths": {
    "registry.example.com": {
      "auth": "base64-encoded-credentials"
    }
  }
}
    Copy to Clipboard Toggle word wrap

For more information, see the Additional resources section.

Build the operating system image with the bootc that contains the Red Hat Edge Manager agent. You can optionally include the following items in your operating system image:

  • The agent configuration for early binding
  • Any drivers
  • Host configuration
  • Application workloads that you need

Complete the following steps:

Procedure

  1. Create a Containerfile file with the following content to build a RHEL 9-based operating system image that includes the Red Hat Edge Manager agent and configuration: 

    FROM registry.redhat.io/rhel9/rhel-bootc:<required_os_version>
    1 
    
RUN dnf --enablerepo ansible-automation-platform-2.5-for-rhel-9-x86_64-rpms -y install flightctl-agent-0.7.2-1.el9fc  && \
    dnf -y clean all && \
    systemctl enable flightctl-agent.service && \
    systemctl mask bootc-fetch-apply-updates.timer
    2
    Copy to Clipboard Toggle word wrap
    1
    The base image that is referenced in FROM is a bootable container (bootc) image that already has a Linux kernel, which allows you to reuse existing standard container build tools and workflows.
    2
    Disables the default automatic updates. The updates are managed by the Red Hat Edge Manager.
    Important

    If your device relies on containers from a private repository, you must place the device pull secret in the /etc/ostree/auth.json path. The pull secret must exist on the device before the secret can be consumed.

    • Optional: To enable podman-compose application support, add the following section to the Containerfile file: 

      RUN dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && \
    dnf -y install podman-compose && \
    dnf -y clean all && \
    systemctl enable podman.service
      Copy to Clipboard Toggle word wrap

    • Optional: If you created the config.yaml for early binding, add the following section to the Containerfile: 

      ADD config.yaml /etc/flightctl/
      Copy to Clipboard Toggle word wrap

    For more information, see Optional: Requesting an enrollment certificate for early binding.

  2. Define the Open Container Initiative (OCI) registry by running the following command: 

    OCI_REGISTRY=registry.redhat.io
    Copy to Clipboard Toggle word wrap

  3. Define the image repository that you have permissions to write to by running the following command: 

    OCI_IMAGE_REPO=${OCI_REGISTRY}/<your_org>/<your_image>
    Copy to Clipboard Toggle word wrap

  4. Define the image tag by running the following command: 

    OCI_IMAGE_TAG=v1
    Copy to Clipboard Toggle word wrap

  5. Build the operating system image for your target platform: 

    sudo podman build -t ${OCI_IMAGE_REPO}:${OCI_IMAGE_TAG} .
    Copy to Clipboard Toggle word wrap

To sign the bootc operating system image by using Sigstore, complete the following steps:

Procedure

  1. Generate a Sigstore key pair named signingkey.pub and signingkey.private: 

    skopeo generate-sigstore-key --output-prefix signingkey
    Copy to Clipboard Toggle word wrap

  2. Configure container tools such as Podman and Skopeo to upload Sigstore signatures together with your signed image to your OCI registry: 

    sudo tee "/etc/containers/registries.d/${OCI_REGISTRY}.yaml" > /dev/null <<EOF
docker:
    ${OCI_REGISTRY}:
        use-sigstore-attachments: true
EOF
    Copy to Clipboard Toggle word wrap

  3. Log in to your OCI registry by running the following command: 

    sudo podman login ${OCI_REGISTRY}
    Copy to Clipboard Toggle word wrap

  4. Sign and publish the operating system image by running the following command: 

    sudo podman push \
    --sign-by-sigstore-private-key ./signingkey.private \
    ${OCI_IMAGE_REPO}:${OCI_IMAGE_TAG}
    Copy to Clipboard Toggle word wrap

4.3.8. Building the operating system disk image
Copy link

Build the operating system disk image that contains the file system for your devices.

Complete the following steps:

Procedure

  1. Create a directory called output by running the following command: 

    mkdir -p output
    Copy to Clipboard Toggle word wrap

  2. Use bootc-image-builder to generate an operating system disk image of type iso from your operating system image by running the following command: 

    sudo podman run --rm -it --privileged --pull=newer \
    --security-opt label=type:unconfined_t \
    -v "${PWD}/output":/output \
    -v /var/lib/containers/storage:/var/lib/containers/storage \
    registry.redhat.io/rhel9/bootc-image-builder:latest \
    --type iso \
    ${OCI_IMAGE_REPO}:${OCI_IMAGE_TAG}
    Copy to Clipboard Toggle word wrap

When the bootc-image-builder completes, you can find the ISO disk image at the ${PWD}/output/bootiso/install.iso path.

Sign and publish your disk image to your Open Container Initiative (OCI) registry. Optionally, you can compress and publish the disk image as an OCI artifact to the same OCI registry as your bootc images, which facilitates a unified hosting and distribution of bootc and disk images. To publish your ISO disk image to a repository named after your bootc image with /diskimage-iso appended.

Prerequisites

Sign and publish your disk image to your OCI registry by completing the following steps:

Procedure

  1. Change the owner of the directory where the ISO disk image is located from root to your current user by running the following command: 

    sudo chown -R $(whoami):$(whoami) "${PWD}/output"
    Copy to Clipboard Toggle word wrap

  2. Define the OCI_DISK_IMAGE_REPO environmental variable to be the same repository as your bootc image with /diskimage-iso appended by running the following command: 

    OCI_DISK_IMAGE_REPO=${OCI_IMAGE_REPO}/diskimage-iso
    Copy to Clipboard Toggle word wrap

  3. Create a manifest list by running the following command: 

    sudo podman manifest create \
    ${OCI_DISK_IMAGE_REPO}:${OCI_IMAGE_TAG}
    Copy to Clipboard Toggle word wrap

  4. Add the ISO disk image to the manifest list as an OCI artifact by running the following command: 

    sudo podman manifest add \
    --artifact --artifact-type application/vnd.diskimage.iso \
    --arch=amd64 --os=linux \
    ${OCI_DISK_IMAGE_REPO}:${OCI_IMAGE_TAG} \
    "${PWD}/output/bootiso/install.iso"
    Copy to Clipboard Toggle word wrap

  5. Sign the manifest list with your private Sigstore key and push the image to the registry by running the following command: 

    sudo podman manifest push --all \
     --sign-by-sigstore-private-key ./signingkey.private \
    ${OCI_DISK_IMAGE_REPO}:${OCI_IMAGE_TAG} \
    docker://${OCI_DISK_IMAGE_REPO}:${OCI_IMAGE_TAG}
    Copy to Clipboard Toggle word wrap

4.3.10. Additional resources
Copy link

4.3.11. Requirements for specific target platforms
Copy link

See the following platform considerations:

When building operating system images and disk images for Red Hat OpenShift Virtualization, you can follow the generic image building process with the following changes:

  • Using late binding by injecting the enrollment certificate or the agent configuration through cloud-init when provisioning the virtual device.
  • Adding the open-vm-tools guest tools to the image.
  • Building a disk image of type qcow2 instead of iso.

Complete the generic steps with changes to the following steps:

Procedure

  1. Build an operating system image based on RHEL 9 that includes the Red Hat Edge Manager agent and VM guest tools but excludes the agent configuration.

  2. Create a file named Containerfile with the following content: 

    FROM registry.redhat.io/rhel9/bootc-image-builder:latest
RUN subscription-manager repos --enable ansible-automation-platform-2.5-for-rhel-9-x86_64-rpms
    dnf -y install flightctl-agent && \
    dnf -y clean all && \
    systemctl enable flightctl-agent.service
RUN dnf -y install cloud-init open-vm-tools && \
    dnf -y clean all && \
    ln -s ../cloud-init.target /usr/lib/systemd/system/default.target.wants && \
    systemctl enable vmtoolsd.service
    Copy to Clipboard Toggle word wrap

  3. Optional: To enable podman-compose application support, add the following section to the Containerfile file: 

    RUN dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && \
    dnf -y install podman-compose && \
    dnf -y clean all && \
    systemctl enable podman.service
    Copy to Clipboard Toggle word wrap
4.3.11.2. Building the bootc image
Copy link

Build, sign, and publish the bootc operating system image by following the generic image building process:

Procedure

  1. Create a directory called output by running the following command: 

    mkdir -p output
    Copy to Clipboard Toggle word wrap

  2. Generate an operating system disk image of type vmdk from your operating system image by running the following command: 

    sudo podman run --rm -it --privileged --pull=newer \
    --security-opt label=type:unconfined_t \
    -v "${PWD}/output":/output \
    -v /var/lib/containers/storage:/var/lib/containers/storage \
    registry.redhat.io/rhel9/bootc-image-builder:latest \
    --type qcow2 \
    ${OCI_IMAGE_REPO}:${OCI_IMAGE_TAG}
    Copy to Clipboard Toggle word wrap

When the bootc-image-builder completes, you can find the disk image under ${PWD}/output/vmdk/disk.vmdk.

4.3.11.3. Building the QCoW2 disk image
Copy link

Red Hat OpenShift Virtualization can download disk images from an OCI registry but it expects a container disk image instead of an OCI artifact.

Complete the following steps to build, sign, and upload the QCoW2 disk image:

Procedure

  1. Create a file called Containerfile.qcow2 with the following content: 

    FROM registry.access.redhat.com/ubi9/ubi:latest AS builder
ADD --chown=107:107 output/qcow2/disk.qcow2 /disk/
    1 
    
RUN chmod 0440 /disk/*
    2 
    
FROM scratch
COPY --from=builder /disk/* /disk/
    3
    Copy to Clipboard Toggle word wrap
    1
    Adds the QCoW2 disk image to a builder container to set the required 107 file ownership, which is the QEMU user.
    2
    Sets the required 0440 file permissions.
    3
    Copies the file to a scratch image.

  2. Build, sign, and publish your disk image by running the following command: 

    sudo chown -R $(whoami):$(whoami) "${PWD}/output"
OCI_DISK_IMAGE_REPO=${OCI_IMAGE_REPO}/diskimage-qcow2
sudo podman build -t ${OCI_DISK_IMAGE_REPO}:${OCI_IMAGE_TAG} -f Containerfile.qcow2 .
sudo podman push --sign-by-sigstore-private-key ./signingkey.private ${OCI_DISK_IMAGE_REPO}:${OCI_IMAGE_TAG}
    Copy to Clipboard Toggle word wrap
4.3.11.4. Building images for VMware vSphere
Copy link

When building operating system images and disk images for VMware vSphere, you can follow the generic image building process with the following changes:

  • Using late binding by injecting the enrollment certificate or the agent configuration through cloud-init when provisioning the virtual device.
  • Adding the open-vm-tools guest tools to the image.
  • Building a disk image of type vmdk instead of iso.

Complete the generic steps with changes to the following steps:

Procedure

  1. Build an operating system image based on RHEL 9 that includes the Red Hat Edge Manager agent and VM guest tools but excludes the agent configuration.

  2. Create a file named Containerfile with the following content: 

    FROM registry.redhat.io/rhel9/bootc-image-builder:latest
RUN subscription-manager repos --enable ansible-automation-platform-2.5-for-rhel-9-x86_64-rpms
    dnf -y install flightctl-agent && \
    dnf -y clean all && \
    systemctl enable flightctl-agent.service && \
RUN dnf -y install cloud-init open-vm-tools && \
    dnf -y clean all && \
    ln -s ../cloud-init.target /usr/lib/systemd/system/default.target.wants && \
    systemctl enable vmtoolsd.service
    Copy to Clipboard Toggle word wrap

  3. Create a directory called output by running the following command: 

    mkdir -p output
    Copy to Clipboard Toggle word wrap

  4. Generate an operating system disk image of type vmdk from your operating system image by running the following command: 

    sudo podman run --rm -it --privileged --pull=newer \
    --security-opt label=type:unconfined_t \
    -v "${PWD}/output":/output \
    -v /var/lib/containers/storage:/var/lib/containers/storage \
    registry.redhat.io/rhel9/bootc-image-builder:latest \
    --type vmdk \
    ${OCI_IMAGE_REPO}:${OCI_IMAGE_TAG}
    Copy to Clipboard Toggle word wrap

When the bootc-image-builder completes, you can find the disk image under ${PWD}/output/vmdk/disk.vmdk.

Chapter 5. Provision devices
Copy link

You can provision devices with the Red Hat Edge Manager in different environments. Use the operating system image or disk image that you built for use with the Red Hat Edge Manager. Depending on your target environment, provision a physical or virtual device.

See the following sections:

5.1. Provision physical devices
Copy link

When you build an International Organization for Standardization (ISO) disk image from an operating system image by using the bootc-image-builder tool, the image is similar to the RHEL ISOs available for download. However, your operating system image content is embedded in the ISO disk image.

To install the ISO disk image to a bare metal system without having access to the network, see Deploying a custom ISO container image in the Red Hat Enterprise Linux documentation.

To install the ISO disk image through the network, see Deploying an ISO bootc image over PXE boot in the Red Hat Enterprise Linux documentation.

You can provision a virtual machine on OpenShift Virtualization by using a QCoW2 container disk image that is hosted on an OCI container registry.

If your operating system image does not already contain the Red Hat Edge Manager agent enrollment configuration, you can inject the configuration through the cloud-init user data at provisioning.

5.2.1. Creating the cloud-init configuration
Copy link

The cloud-init configuration customizes a virtual machine instance on its first boot, allowing you to automatically enroll it as a new agent in your Red Hat Edge Manager service.

Prerequisites

  • You installed the flightctl CLI and logged in to your Red Hat Edge Manager service instance.
  • You installed the oc CLI, used it to log in to your OpenShift cluster instance, and changed to the project in which you want to create your virtual machine.

Procedure

  1. Request a new Red Hat Edge Manager agent enrollment configuration and store it in a file called config.yaml by running the following command: 

    flightctl certificate request --signer=enrollment --expiration=365d --output=embedded > config.yaml
    Copy to Clipboard Toggle word wrap

  2. Create a cloud configuration user data file called cloud-config.yaml that places the agent configuration in the correct location on the first boot by running the following command: 

    cat <<EOF > cloud-config.yaml
#cloud-config
write_files:
- path: /etc/flightctl/config.yaml
  content: $(cat config.yaml | base64 -w0)
  encoding: b64
EOF
    Copy to Clipboard Toggle word wrap

  3. Create a Kubernetes Secret that contains the cloud configuration user data file: 

    oc create secret generic enrollment-secret --from-file=userdata=cloud-config.yaml
    Copy to Clipboard Toggle word wrap

5.2.2. Creating the virtual machine
Copy link

Create a virtual machine that has its primary disk populated from your QCoW2 container disk image and a cloud-init configuration drive that is populated from your enrollment secret.

Complete the following steps:

Procedure

  1. Create a file that has the VirtualMachine resource manifest by running the following command: 

    cat <<EOF > my-bootc-vm.yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: my-bootc-vm
spec:
  runStrategy: RerunOnFailure
  template:
    spec:
      domain:
        cpu:
          cores: 1
        memory:
          guest: 1024M
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
      volumes:
        - name: containerdisk
          containerDisk:
            image: ${OCI_DISK_IMAGE_REPO}:${OCI_IMAGE_TAG}
        - name: cloudinitdisk
          cloudInitConfigDrive:
            secretRef:
              name: enrollment-secret
EOF
    Copy to Clipboard Toggle word wrap

  2. Apply the resource manifest to your cluster by running the following command: 

    oc apply -f my-bootc-vm.yaml
    Copy to Clipboard Toggle word wrap

Additional resources

Chapter 6. Manage devices
Copy link

The Red Hat Edge Manager manages the device lifecycle from enrollment to decommissioning of a device. The device lifecycle also includes device management, such as organizing, monitoring, and updating your devices with the Red Hat Edge Manager.

You can manage your devices individually or in a fleet. With the Red Hat Edge Manager you can manage a whole fleet of devices as a single object instead of managing many devices individually.

You only need to specify the required configuration once, and then the Red Hat Edge Manager applies the configuration to all devices in the fleet.

Understanding individual device management is the foundation for managing devices in a fleet. You might want to manage your devices individually in the following scenarios:

  • If a few devices have different configurations.
  • If you use external automation for updating the device.

The following sections focus on managing individual devices:

6.1. Enroll devices
Copy link

To manage your devices with the Red Hat Edge Manager, you must enroll the devices to the Red Hat Edge Manager service.

The first time the Red Hat Edge Manager agent runs on a device, the agent prepares for the enrollment process by generating a cryptographic key pair. The cryptographic key pair serves as the unique cryptographic identity of the device. The key pair consists of a public and a private key. The private key never leaves the device, so that the device cannot be duplicated or impersonated.

When the device is not yet enrolled, the agent performs service discovery to find its Red Hat Edge Manager service instance. Then, the device establishes a secure, mTLS-protected network connection to the service. The device uses its X.509 enrollment certificate that the device acquired during image building or device provisioning. The device submits an enrollment request to the service that includes the following:

  • a description of the device hardware and operating system
  • an X.509 Certificate Signing Request which includes the cryptographic identity of the device to obtain the initial management certificate

The device is not considered trusted and remains quarantined in a device lobby until an authorized user approves or denies the request.

For more information, see the following sections:

6.1.1. Enrolling devices on the CLI
Copy link

You must enroll devices into the Red Hat Edge Manager service before you can manage them.

Prerequisites

Procedure

  1. List all devices that are currently waiting for approval by running the following command: 

    flightctl get enrollmentrequests --field-selector="status.approval.approved != true"
    Copy to Clipboard Toggle word wrap

    See the following example: 

    NAME           APPROVAL  APPROVER  APPROVED LABELS
<device_name>  Pending   <none>    <none>
    Copy to Clipboard Toggle word wrap
    Note

    The unique device name is generated by the agent and you cannot change it. The agent chooses a base32-encoded hash of its public key as the device name.

  2. Approve an enrollment request by specifying the name of the enrollment request. Optionally, you can add labels to the device by using the --label or -l flags. See the following example: 

    flightctl approve -l region=eu-west-1 -l site=factory-berlin enrollmentrequest/54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg
    Copy to Clipboard Toggle word wrap

    See the following example output: 

    NAME           APPROVAL  APPROVER  APPROVED LABELS
<device_name>  Approved  user      region=eu-west-1,site=factory-berlin
    Copy to Clipboard Toggle word wrap

After you approve the enrollment request, the service issues the management certificate for the device and registers the device in the device inventory. You can then manage the device.

6.2. View devices
Copy link

To get more information about the devices in your inventory, you can use the Red Hat Edge Manager CLI.

You can view details for enrolled devices, including their status and health, on the Red Hat Edge Manager web UI

Prerequisites

Procedure

  1. From the navigation panel, select Application Links → Edge Manager. This opens the external Edge Manager instance.
  2. From the navigation panel, select Devices where you can view your device inventory, details, and decommission devices.

Complete the following steps:

Procedure

  1. View the devices in the device inventory by running the following command: 

    flightctl get devices
    Copy to Clipboard Toggle word wrap

    See the following example output: 

    NAME           ALIAS    OWNER   SYSTEM  UPDATED     APPLICATIONS  LAST SEEN
<device_name>  <none>   <none>  Online  Up-to-date  <none>        3 seconds ago
    Copy to Clipboard Toggle word wrap

  2. View the details of this device in YAML format by running the following command: 

    flightctl get device/<device_name> -o yaml
    Copy to Clipboard Toggle word wrap

    See the following example output: 

    apiVersion: flightctl.io/v1alpha1
kind: Device
metadata:
  name: <device_name>
  labels:
    1 
    
    region: eu-west-1
    site: factory-berlin
spec:
  os:
    image: quay.io/flightctl/rhel:9.5
    2 
    
  config:
  - name: my-os-configuration
    3 
    
    configType: GitConfigProviderSpec
    gitRef:
      path: /configuration
      repository: my-configuration-repo
      targetRevision: production
status:
  os:
    image: quay.io/flightctl/rhel:9.5
    4 
    
  config:
    renderedVersion: "1"
    5 
    
  applications:
    data: {}
    6 
    
    summary:
      status: Unknown
    7 
    
  resources:
    8 
    
    cpu: Healthy
    disk: Healthy
    memory: Healthy
  systemInfo:
    9 
    
    architecture: amd64
    bootID: 037750f7-f293-4c5b-b06e-481eef4e883f
    operatingSystem: linux
  summary:
    info: ""
    status: Online
    10 
    
  updated:
    status: UpToDate
    11 
    
  lastSeen: "2024-08-28T11:45:34.812851905Z"
    12 
    
[...]
    Copy to Clipboard Toggle word wrap
    1
    User-defined labels assigned to the device.
    2
    The target OS image version of the device.
    3
    The target OS configuration of the device.
    4
    The current OS image version of the device
    5
    The current OS configuration version of the device.
    6
    The current list of deployed applications of the device.
    7
    The health status of applications on the device.
    8
    The availability of CPU, disk, and memory resources.
    9
    Basic system information.
    10
    The health status of the device.
    11
    The update status of the device.
    12
    The last check-in time and date of the device.

6.2.3. Labels and label selectors
Copy link

You can organize your resources by assigning them labels, for example, to record their location, hardware type, or purpose. The Red Hat Edge Manager labels follow the same syntax, principles, and operators as Kubernetes labels and label selectors. You can select devices with labels when viewing the device inventory or applying operations to the devices.

Labels follow the key=value format. You can use the key to group devices. For example, if your labels follow the site=<location> naming convention, you can group your devices by site. You can also use labels that only consist of keys.

Labels must adhere to the following rules to be valid:

  • Keys and value must each be 63 characters or less.
  • Keys and values can consist of alphanumeric characters (a-z, A-Z, 0-9).
  • Keys and values can also contain dashes (-), underscores (_), dots (.) but not as the first or last character.
  • Value can be omitted.

You can apply labels to devices in the following ways:

  • Define a set of default labels during image building that are automatically applied to all devices during deployment.
  • Assign initial labels during enrollment.
  • Assign labels post-enrollment.

When resources are labeled, you can select a subset of devices by creating a label selector. A label selector is a comma-separated list of labels for selecting devices that have the same set of labels.

See the following examples:

Expand
Example label selectorSelected devices

site=factory-berlin

All devices with a site label key and a factory-berlin label value.

site!=factory-berlin

All devices with a site label key but where the label value is not factory-berlin.

site in (factory-berlin,factory-madrid)

All devices with a site label key and where the label value is either factory-berlin or factory-madrid.

For more information about labels and selectors, see Labels and Selectors in the Kubernetes documentation.

View devices and their associated labels on the web UI. You can use labels to organize your devices and device fleets.

Complete the following steps:

  1. From the navigation panel, select Application Links → Edge Manager. This opens the external Edge Manager instance.
  2. From the navigation panel, select Devices.
  3. Select the device you want to manage. In the Details tab you can view the associated labels under Labels.

View devices and their associated labels. You can use labels to organize your devices and device fleets.

Complete the following steps:

Procedure

  1. View devices in your inventory with their labels by using the -o wide option: 

    flightctl get devices -o wide
    Copy to Clipboard Toggle word wrap

    See the following example output: 

    NAME            ALIAS    OWNER   SYSTEM  UPDATED     APPLICATIONS  LAST SEEN      LABELS
<device1_name>  <none>   <none>  Online  Up-to-date  <none>        3 seconds ago  region=eu-west-1,site=factory-berlin
<device2_name>  <none>   <none>  Online  Up-to-date  <none>        1 minute ago   region=eu-west-1,site=factory-madrid
    Copy to Clipboard Toggle word wrap

  2. View devices in your inventory with a specific label or set of labels by using the -l <key=value> option: 

    flightctl get devices -l site=factory-berlin -o wide
    Copy to Clipboard Toggle word wrap

    See the following example output: 

    NAME            ALIAS    OWNER   SYSTEM  UPDATED     APPLICATIONS  LAST SEEN      LABELS
<device1_name>  <none>   <none>  Online  Up-to-date  <none>        3 seconds ago  region=eu-west-1,site=factory-berlin
    Copy to Clipboard Toggle word wrap
6.2.3.3. Updating labels on the CLI
Copy link

Update labels on your devices by using the CLI.

Complete the following steps:

Procedure

  1. Export the current definition of the device into a file by running the following command: 

    flightctl get device/<device1_name> -o yaml > my_device.yaml
    Copy to Clipboard Toggle word wrap

  2. Use your preferred editor to edit the my_device.yaml file. See the following example: 

    apiVersion: flightctl.io/v1alpha1
kind: Device
metadata:
  labels:
    some_key: some_value
    some_other_key: some_other_value
  name: <device1_name>
spec:
[...]
    Copy to Clipboard Toggle word wrap

  3. Save the file and apply the updated device definition by running the following command: 

    flightctl apply -f my_device.yaml
    Copy to Clipboard Toggle word wrap

  4. Verify your changes by running the following example output: 

    NAME            ALIAS    OWNER   SYSTEM  UPDATED     APPLICATIONS  LAST SEEN      LABELS
<device1_name>  <none>   <none>  Online  Up-to-date  <none>        3 minutes ago  some_key=some_value,some_other_key=some_other_value
<device2_name>  <none>   <none>  Online  Up-to-date  <none>        4 minutes ago  region=eu-west-1,site=factory-madrid
    Copy to Clipboard Toggle word wrap

6.2.4. Field selectors
Copy link

Field selectors filter a list of Red Hat Edge Manager resources based on specific resource field values. They follow the same syntax, principles, and operators as Kubernetes Field and Label selectors, with additional operators available for more advanced search use cases.

6.2.4.1. Supported fields
Copy link

Red Hat Edge Manager resources give a set of metadata fields that you can select.

Each resource supports the following metadata fields:

  • metadata.name
  • metadata.owner
  • metadata.creationTimestamp
Note

To query labels, use Label Selectors for advanced and flexible label filtering.

For more information, see Labels and label selectors.

6.2.4.2. List of additional supported fields
Copy link

In addition to the metadata fields, each resource has its own unique set of fields that you can select, offering further flexibility in filtering and selection based on resource-specific attributes.

The following table lists the fields supported for filtering for each resource kind:

Expand
KindFields

Certificate Signing Request

status.certificate

Device

status.summary.status

status.applicationsSummary.status

status.updated.status

status.lastSeen

status.lifecycle.status

Enrollment Request

status.approval.approved

status.certificate

Fleet

spec.template.spec.os.image

Repository

spec.type

spec.url

Resource Sync

spec.repository

6.2.4.3. Fields discovery
Copy link

Some Red Hat Edge Manager resources might expose additional supported fields. You can discover the supported fields by using flightctl with the --field-selector option. If you try to use an unsupported field, the error message lists the available supported fields.

See the following examples: 

flightctl get device --field-selector='text'
Copy to Clipboard Toggle word wrap 
Error: listing devices: 400, message: unknown or unsupported selector: unable to resolve selector name "text". Supported selectors are: [metadata.alias metadata.creationTimestamp metadata.name metadata.nameoralias metadata.owner status.applicationsSummary.status status.lastSeen status.summary.status status.updated.status]
Copy to Clipboard Toggle word wrap

The field text is not a valid field for filtering. The error message provides a list of supported fields that you can use with --field-selector for the Device resource.

You can then use one of the supported fields: 

flightctl get devices --field-selector 'metadata.alias contains cluster'
Copy to Clipboard Toggle word wrap

The metadata.alias field is checked with the containment operator contains to see if it has the value cluster.

Examples

Example 1: Excluding a specific device by name

The following command filters out a specific device by its name: 

flightctl get devices --field-selector 'metadata.name!=c3tkb18x9fw32fzx5l556n0p0dracwbl4uiojxu19g2'
Copy to Clipboard Toggle word wrap

Example 2: Filter by owner, labels, and creation timestamp

This command retrieves devices owned by Fleet/pos-fleet, located in the us region, and created in 2024: 

flightctl get devices --field-selector 'metadata.owner=Fleet/pos-fleet, metadata.creationTimestamp >= 2024-01-01T00:00:00Z, metadata.creationTimestamp < //2025-01-01T00:00:00Z' -l 'region=us'
Copy to Clipboard Toggle word wrap

Example 3: Filter by Owner, Labels, and Device Status

This command retrieves devices owned by Fleet/pos-fleet, located in the us region, and with a status.updated.status of either Unknown or OutOfDate: 

flightctl get devices --field-selector 'metadata.owner=Fleet/pos-fleet, status.updated.status in (Unknown, OutOfDate)' -l 'region=us'
Copy to Clipboard Toggle word wrap
6.2.4.4. Supported operators
Copy link
Expand
OperatorSymbolDescription

Exists

exists

Checks if a field exists

DoesNotExist

!

Checks if a field does not exist

Equals

=

Checks if a field is equal to a value

DoubleEquals

==

Another form of equality check

NotEquals

!=

Checks if a field is not equal to a value

GreaterThan

>

Checks if a field is greater than a value

GreaterThanOrEquals

>=

Checks if a field is greater than or equal to a value

LessThan

<

Checks if a field is less than a value

LessThanOrEquals

⇐

Checks if a field is less than or equal to a value

In

in

Checks if a field is within a list of values

NotIn

notin

Checks if a field is not in a list of values

Contains

contains

Checks if a field has a value

NotContains

notcontains

Checks if a field does not contain a value

6.2.4.4.1. Operators usage by field type
Copy link

Each field type supports a specific subset of operators:

Expand
Field TypeSupported OperatorsValue

String

Equals: Matches if the field value is an exact match to the specified string.

DoubleEquals: Matches if the field value is an exact match to the specified string (alternative to Equals).

NotEquals: Matches if the field value is not an exact match to the specified string.

In: Matches if the field value matches at least one string in the list.

NotIn: Matches if the field value does not match any of the strings in the list.

Contains: Matches if the field value has the specified substring.

NotContains: Matches if the field value does not contain the specified substring.

Exists: Matches if the field is present.

DoesNotExist: Matches if the field is not present.

Text string

Timestamp

Equals: Matches if the field value is an exact match to the specified timestamp.

DoubleEquals: Matches if the field value is an exact match to the specified timestamp (alternative to Equals).

NotEquals: Matches if the field value is not an exact match to the specified timestamp.

GreaterThan: Matches if the field value is after the specified timestamp.

GreaterThanOrEquals: Matches if the field value is after or equal to the specified timestamp.

LessThan: Matches if the field value is before the specified timestamp.

LessThanOrEquals: Matches if the field value is before or equal to the specified timestamp.

In: Matches if the field value matches at least one timestamp in the list.

NotIn: Matches if the field value does not match any of the timestamps in the list.

Exists: Matches if the field is present.

DoesNotExist: Matches if the field is not present.

RFC 3339 format

Number

Equals: Matches if the field value equals the specified number.

DoubleEquals: Matches if the field value equals the specified number (alternative to Equals).

NotEquals: Matches if the field value does not equal to the specified number.

GreaterThan: Matches if the field value is greater than the specified number.

GreaterThanOrEquals: Matches if the field value is greater than or equal to the specified number.

LessThan: Matches if the field value is less than the specified number.

LessThanOrEquals: Matches if the field value is less than or equal to the specified number.

In: Matches if the field value equals at least one number in the list.

NotIn: Matches if the field value does not equal any numbers in the list.

Exists:Matches if the field is present.

DoesNotExist: Matches if the field is not present.

Number format

Boolean

Equals: Matches if the value is true or false.

DoubleEquals: Matches if the value is true or false (alternative to Equals).

NotEquals: Matches if the value is the opposite of the specified value.

In: Matches if the value (true or false) is in the list.

Note

The list can only contain true or false, so this operator is limited in use.

NotIn: Matches if the value is not in the list.

Exists: Matches if the field is present.

DoesNotExist: Matches if the field is not present.

Boolean format (true, false)

Array

Contains: Matches if the array has the specified value.

NotContains: Matches if the array does not contain the specified value. In: Matches if the array overlaps with the specified values.

NotIn: Matches if the array does not overlap with the specified values. Exists: Matches if the field is present.

DoesNotExist:Matches if the field is not present.

Note

Using Array[Index] treats the element as the type defined for the array elements. For example string, timestamp, number, or boolean.

Array element

6.3. Update the operating system
Copy link

You can update the operating system of a device by updating the target operating system image name or version in the device specification. When the agent communicates with the server, the agent detects the requested update. Then, the agent automatically starts downloading and verifying the new operating system version in the background. The Red Hat Edge Manager agent schedules the actual system update that is performed according to the update policy. At the scheduled update time, the agent installs the new version without disrupting the currently running operating system. Finally, the device reboots into the new version.

The Red Hat Edge Manager currently supports the following image type and image reference format:

Expand
Image TypeImage Reference

bootc

An OCI image reference to a container registry. Example: quay.io/flightctl-example/rhel:9.5

During the process, the agent sends status updates to the service. You can check the update process by viewing the device status.

For more information, see View devices.

6.3.1. Updating the operating system on the CLI
Copy link

Update a device using the CLI.

Complete the following steps:

Procedure

  1. Get the current resource manifest of the device by running the following command: 

    flightctl get device/<device_name> -o yaml > my_device.yaml
    Copy to Clipboard Toggle word wrap

  2. Edit the Device resource to specify the new operating system name and version target. 

    apiVersion: flightctl.io/v1alpha1
kind: Device
metadata:
  name: <device_name>
spec:
[...]
  os:
    image: quay.io/flightctl/rhel:9.5
[...]
    Copy to Clipboard Toggle word wrap

  3. Apply the updated Device resource by running the following command: 

    flightctl apply -f <device_name>.yaml
    Copy to Clipboard Toggle word wrap

You can include an operating system-level host configuration in the image to give maximum consistency and repeatability. To update the configuration, create a new operating system image and update devices with the new image.

However, updating devices with a new image can be impractical in the following cases:

  • The configuration is missing in the image.
  • The configuration needs to be specific to a device.
  • The configuration needs to be updateable at runtime without updating the operating system image and rebooting.

For these cases, you can declare a set of configuration files that are present on the file system of the device. The Red Hat Edge Manager agent applies updates to the configuration files while ensuring that either all files are successfully updated in the file system, or rolled back to their pre-update state. If the user updates both an operating system and configuration set of a device at the same time, the Red Hat Edge Manager agent updates the operating system first. It then applies the specified set of configuration files.

You can also specify a list of configuration sets that the Red Hat Edge Manager agent applies in sequence. In case of a conflict, the last applied configuration set is valid.

Important

After the Red Hat Edge Manager agent updates the configuration on the disk, the running applications need to reload the new configuration into memory for the configuration to become effective. If the update involves a reboot, systemd automatically restarts the applications with the new configuration and in the correct order. If the update does not involve a reboot, many applications can detect changes to their configuration files and automatically reload the files. When an application does not support change detection, you can use device lifecycle hooks to run scripts or commands if certain conditions are met.

6.4.1. Configuration providers
Copy link

You can provide configuration from many sources, called configuration providers, in Red Hat Edge Manager. The Red Hat Edge Manager currently supports the following configuration providers:

Git Config Provider
Fetches device configuration files from a Git repository.
Kubernetes Secret Provider
Fetches a secret from a Kubernetes cluster and writes the content to the file system of the device.
HTTP Config Provider
Fetches device configuration files from an HTTP(S) endpoint.
Inline Config Provider
Allows specifying device configuration files inline in the device manifest without querying external systems.

Read more about the configuration providers in the following sections:

6.4.1.1. Configuration from a Git repository
Copy link

You can store device configuration in a Git repository such as GitHub or GitLab. You can then add a Git Config Provider so that the Red Hat Edge Manager synchronizes the configuration from the repository to the file system of the device.

The Git Config Provider takes the following parameters:

Expand

Parameter

Description

Repository

The name of a Repository resource defined in the Red Hat Edge Manager.

TargetRevision

The branch, tag, or commit of the repository to checkout.

Path

The absolute path to the directory in the repository from which files and subdirectories are synchronized to the file system of the device. The Path directory corresponds to the root directory (/) on the device, unless you specify the MountPath parameter.

MountPath

Optional. The absolute path to the directory in the file system of the device to write the content of the repository to. By default, the value is the file system root (/).

The Repository resource defines the Git repository, the protocol, and the access credentials that the Red Hat Edge Manager must use. You only need to set up the repository once. After setting up, you can use the repository to configure individual devices or device fleets.

6.4.1.2. Secrets from a Kubernetes cluster
Copy link

The Red Hat Edge Manager can query only the Kubernetes cluster that the Red Hat Edge Manager is running on for a Kubernetes secret. You can write the content of that secret to a path on the device file system.

The Kubernetes Secret Provider takes the following parameters:

Expand

Parameter

Description

Name

The name of the secret.

NameSpace

The namespace of the secret.

MountPath

The directory in the file system of the device to write the secret contents to.

Note

The Red Hat Edge Manager needs permission to access secrets in the defined namespace. For example, creating a ClusterRole and ClusterRoleBinding allows the flightctl-worker service account to get and list secrets in that namespace.

6.4.1.3. Configuration from an HTTP server
Copy link

The Red Hat Edge Manager can query an HTTP server for configuration. The HTTP server can serve static or dynamically generated configuration for a device.

The HTTP Config Provider takes the following parameters:

Expand

Parameter

Description

Repository

The name of a Repository resource defined in the Red Hat Edge Manager.

Suffix

The suffix to append to the base URL defined in the Repository resource. The suffix can include path and query parameters, for example /path/to/endpoint?query=param.

FilePath

The absolute path to the file in the file system of the device to write the response of the HTTP server to.

The Repository resource specifies the HTTP server for the Red Hat Edge Manager to connect to, and the protocol and access credentials to use. You must set up the repository needs once, and then you can use the repository to configure many devices or device fleets.

You can specify configuration inline in a device specification. When you use the inline device specification, the Red Hat Edge Manager does not need to connect to external systems to fetch the configuration.

The Inline Config Provider takes a list of file specifications, where each file specification takes the following parameters:

Expand

Parameter

Description

Path

The absolute path to the file in the file system of the device to write the content to. If a file already exists in the specified path, the file is overwritten.

Content

The UTF-8 or base64-encoded content of the file.

ContentEncoding

Defines how the contents are encoded. Must be either plain or base64. Default value is set to plain.

Mode

Optional. The permission mode of the file. You can specify the octal with a leading zero, for example 0644, or as a decimal without a leading zero, for example 420. The setuid, setgid, and sticky bits are supported. If not specified, the permission mode for files defaults to 0644.

User

Optional. The owner of the file. Specified either as a name or numeric ID. Default value is set to root.

Group

Optional. The group of the file. Specified either as a name or numeric ID.

Additional resources

Create and apply a device configuration in a Git repository.

Complete the following steps:

Procedure

  1. Create a file, for example site-settings-repo.yaml, that contains the following definition for a Repository resource, named site-settings: 

    apiVersion: flightctl.io/v1alpha1
kind: Repository
metadata:
  name: site-settings
spec:
  type: git
  url: https://github.com/<your_org>/<your_repo>.git
    Copy to Clipboard Toggle word wrap

  2. Create the Repository resource by running the following command: 

    flightctl apply -f site-settings-repo.yaml
    Copy to Clipboard Toggle word wrap

  3. Verify that the resource has been correctly created and is accessible by Red Hat Edge Manager running the following command: 

    flightctl get repository/site-settings
    Copy to Clipboard Toggle word wrap

    See the following example output: 

    NAME           TYPE  REPOSITORY URL                                 ACCESSIBLE
site-settings  git   https://github.com/<your_org>/<your_repo>.git  True
    Copy to Clipboard Toggle word wrap

  4. Apply the example-site configuration to a device by updating the device specification: 

    apiVersion: flightctl.io/v1alpha1
kind: Device
metadata:
  name: <device_name>
spec:
[...]
  config:
    1 
    
  - name: example-site
    configType: GitConfigProviderSpec
    gitRef:
      repository: site-settings
      targetRevision: production
      path: /etc/example-site
    2 
    
[...]
    Copy to Clipboard Toggle word wrap
    1
    The example configuration takes all the files from the example-site directory from the production branch of the site-settings repository and places the files in the root directory (/).
    2
    Ensure that the target path is writeable by creating your directory structure. The root directory (/) is not writeable in bootc systems.

6.5. Use device lifecycle hooks
Copy link

The Red Hat Edge Manager agent can run user-defined commands at specific points in the device lifecycle by using device lifecycle hooks. For example, you can add a shell script to your operating system images that backs up your application data. You can then specify that the script must run and complete successfully before the agent can start updating the operating system.

As another example, certain applications or system services do not automatically reload their configuration file when the file changes on the disk. You can manually reload the configuration file by specifying a command as another hook, which is called after the agent completes the update process.

The following device lifecycle hooks are supported:

Expand
Lifecycle HookDescription

beforeUpdating

This hook is called after the agent completed preparing for the update and before actually making changes to the system. If an action in this hook returns with failure, the agent cancels the update.

afterUpdating

This hook is called after the agent has written the update to disk. If an action in this hook returns with failure,the agent cancels and rolls back the update.

beforeRebooting

This hook is called before the system reboots. The agent blocks the reboot until running the action has completed or timed out. If any action in this hook returns with failure, the agent cancels and rolls back the update.

afterRebooting

This hook is called when the agent first starts after a reboot. If any action in this hook returns with failure, the agent reports this but continues starting up.

6.5.1. Rule files
Copy link

You can define device lifecycle hooks by adding rule files to one of the following locations in the device file system:

  • Rules in the /usr/lib/flightctl/hooks.d/<lifecycle_hook_name>/ drop-in directory are read-only. To add rules to the /usr directory, you must add them to the operating system image during image building.
  • Rules in the /etc/flightctl/hooks.d/<lifecycle_hook_name>/ drop-in directory are read-writable. You can update the rules at runtime by using several methods.

When creating and placing the files, you must consider the following practices:

  • The name of the rule must be all lower case.
  • If you define rules in both locations, the rules are merged.
  • If you add more than one rule files to a lifecycle hook directory, the files are processed in lexical order of the file names.
  • If you define files with identical file names in both locations, the file in the /etc folder takes precedence over the file of the same name in the /usr folder.

A rule file is written in YAML format and has a list of one or more actions. An action can be an instruction to run an external command.

When you specify many actions for a hook, the actions are performed in sequence, finishing one action before starting the next.

If an action returns with a failure, the following actions are skipped.

A run action takes the following parameters:

Expand

Parameter

Description

Run

The absolute path to the command to run, followed by any flags or arguments, for example /usr/bin/nmcli connection reload. The command is not executed in a shell, so you cannot use shell variables, such as $PATH or $HOME, or chain commands, such as | or ;. If necessary, you can start a shell by specifying the shell as command to run, for example /usr/bin/bash -c 'echo $SHELL $HOME $USER'.

EnvVars

Optional. A list of key-value pairs to set as environment variables for the command.

WorkDir

Optional. The directory the command is run from.

Timeout

Optional. The maximum duration that is allowed for the action to complete. Specify the duration as a single positive integer followed by a time unit. The s, m, and h units are supported for seconds, minutes, and hours.

If

Optional. A list of conditions that must be true for the action to be run. If not provided, actions run unconditionally.

By default, the system performs actions every time the hook is triggered. However, for the afterUpdating hook, you can use the If parameter to add conditions that must be true for an action to be performed. Otherwise, the action is skipped.

For example, to run an action only if a given file or directory changes during the update, you can define a path condition that takes the following parameters:

Expand

Parameter

Description

Job type

An absolute path to a file or directory that must change during the update as a condition for the action to be performed. Specify paths by using forward slashes (/):

  • If the path is to a directory, it must end with a forward slash (/).
  • If you specify a path to a file, the file must have changed to satisfy the condition
  • If you specify a path to a directory, a file in that directory or any of its subdirectories must have changed to satisfy the condition

Op

A list of file operations, such as created, updated, and removed, to limit the type of changes to the specified path as a condition for the action to be performed.

If you specify a path condition for an action in the afterUpdating hook, you have the following variables that you can include in arguments to your command and are replaced with the absolute paths to the changed files:

Expand

Variable

Description

${ Path }

The absolute path to the file or directory specified in the path condition.

${ Files }

A space-separated list of absolute paths of the files that changed during the update and are covered by the path condition.

${ CreatedFiles }

A space-separated list of absolute paths of the files that were created during the update and are covered by the path condition.

${ UpdatedFiles }

A space-separated list of absolute paths of the files that were updated during the update and are covered by the path condition.

${ RemovedFiles }

A space-separated list of absolute paths of the files that were removed during the update and are covered by the path condition.

The Red Hat Edge Manager agent includes a built-in set of rules defined in /usr/lib/flightctl/hooks.d/afterupdating/00-default.yaml. The following commands are executed if certain files are changed:

Expand

File

Command

Description

/etc/systemd/system/

systemctl daemon-reload

Changes to systemd units are activated by signaling the systemd daemon to reload the systemd manager configuration. This reruns all generators, reloads all unit files, and re-creates the entire dependency tree.

/etc/NetworkManager/system-connections/

nmcli conn reload

Changes to NetworkManager system connections are activated by signaling the NetworkManager daemon to reload all connections. For more information, see the Additional resources section.

/etc/firewalld/

firewall-cmd --reload

Changes to the permanent configuration of firewalld are activated by signaling firewalld to reload firewall rules as new runtime configuration.

Additional resources

6.6. Monitor device resources
Copy link

You can set up monitors for device resources and define alerts when the use of these resources crosses a defined threshold. When the agent alerts the Red Hat Edge Manager service, the service sets the device status to "degraded" or "error" (depending on the severity level).

Resource monitors take the following parameters:

Expand
ParameterDescription

MonitorType

The resource to monitor. Currently supported resources are "CPU", "Memory", and "Disk".

SamplingInterval

The interval in which the monitor samples use, specified as positive integer followed by a time unit ("s" for seconds, "m" for minutes, "h" for hours).

AlertRules

A list of alert rules.

Path

(Disk monitor only) The absolute path to the directory to monitor. Utilization reflects the filesystem containing the path, similar to df, even if it’s not a mount point.

Alert rules take the following parameters:

Expand
ParameterDescription

Severity

The alert rule’s severity level out of "Info", "Warning", or "Critical". Only one alert rule is allowed per severity level and monitor.

Duration

The duration that resource use is measured and averaged over when sampling, specified as positive integer followed by a time unit ("s" for seconds, "m" for minutes, "h" for hours). It must be smaller than the sampling interval.

Percentage

The use threshold that triggers the alert, as percentage value (range 0 to 100 without the "%" sign).

Description

A human-readable description of the alert. This is useful for adding details about the alert that might help with debugging. By default it populates the alert as : load is above >% for more than.

6.6.1. Monitoring device resources on the CLI
Copy link

Monitor the resources of your device through the CLI, providing you with the tools and commands to track performance and troubleshoot issues.

Procedure

  • Add resource monitors in the resources: section of the device’s specification.

    For example, add the following monitor for your disk: 

    apiVersion: flightctl.io/v1alpha1
kind: Device
metadata:
  name: <device_name>
spec:
[...]
  resources:
  - monitorType: Disk
    samplingInterval: 5s
    1 
    
    path: /application_data
    2 
    
    alertRules:
    - severity: Warning
    3 
    
      duration: 30m
      percentage: 75
      description: Disk space for application data is >75% full for over 30m.
    - severity: Critical
    4 
    
      duration: 10m
      percentage: 90
      description: Disk space for application data is >90% full over 10m.
[...]
    Copy to Clipboard Toggle word wrap
    1
    Samples usage every 5 seconds.
    2
    Checks disk usage on the file system that is associated with the /applications_data path.
    3
    Initiates a warning if the average usage exceeds 75% for more than 30 minutes.
    4
    Initiates a critical alert if the average usage exceeds 90% for over 10 minutes.

You can deploy, update, or remove applications on a device by updating the list of applications in the device specification. When the Red Hat Edge Manager agent checks in and detects the change in the specification, the agent downloads any new or updated application packages and images from an Open Container Initiative (OCI)-compatible registry. Then, the agent deploys the packages to the appropriate application runtime or removes them from that runtime.

The Red Hat Edge Manager supports the podman-compose tool as the application runtime and format.

For more information, see Building a bootc operating system image for use with the Red Hat Edge Manager.

7.1. Building an application package image
Copy link

The Red Hat Edge Manager can download application packages from an Open Container Initiative (OCI) compatible registry. You can build an OCI container image that includes your application package in the podman-compose format and push the image to your OCI registry.

Prerequisites

  • You must install the Red Hat Edge Manager CLI.
  • You must log in to the Red Hat Edge Manager service.
  • Your device must run an operating system image with the podman-compose tool installed.

Procedure

  1. Define the functionality of the application in a file called podman-compose.yaml that follows the Podman Compose specification:

    • Create a file called Containerfile with the following content: 

      FROM scratch
      1 
      
COPY podman-compose.yaml /podman-compose.yaml
LABEL appType="compose"
      2
      Copy to Clipboard Toggle word wrap
      1
      Embed the compose file in a scratch container.
      2
      Add the appType=compose label.

  2. Build and push the container image to your OCI registry:

    1. Define the image repository that you have permissions to write to by running the following command: 

      OCI_IMAGE_REPO=quai.io/<your_org>/<your_image>
      Copy to Clipboard Toggle word wrap

    2. Define the image tag by running the following command: 

      OCI_IMAGE_TAG=v1
      Copy to Clipboard Toggle word wrap

    3. Build the application container image by running the following command: 

      podman build -t ${OCI_IMAGE_REPO}:${OCI_IMAGE_TAG} .
      Copy to Clipboard Toggle word wrap

    4. Push the container image by running the following command: 

      podman push ${OCI_IMAGE_REPO}:${OCI_IMAGE_TAG} .
      Copy to Clipboard Toggle word wrap

Application manifests are specified inline in a device’s specification, so you do not need to build an OCI registry application package.

The inline application provider accepts a list of application content with the following parameters:

Expand

Parameter

Description

Path

The relative path to the file on the device. Note that any existing file is overwritten.

Content (Optional)

The plain text (UTF-8) or base64-encoded content of the file.

ContentEncoding

How the contents are encoded. Must be either "plain" or "base64". Defaults to "plain".

Example 

apiVersion: flightctl.io/v1alpha1
kind: Device
metadata:
  name: some_device_name
spec:
[...]
  applications:
    - name: my-app
      appType: compose
      inline:
        - content: |
            version: "3.8"
            services:
              service1:
                image:  quay.io/flightctl-tests/alpine:v1
                command: ["sleep", "infinity"]
          path: podman-compose.yaml
[...]
Copy to Clipboard Toggle word wrap
Note

Inline compose applications can have two paths at most. You must name the first one podman-compose.yaml, and the second (override) podman-compose.override.yaml.

Deploy an application package to a device from an OCI registry by using the CLI.

Complete the following steps:

Procedure

  1. Specify the application package that you want to deploy in the spec.applications field in the Device resource: 

    apiVersion: flightctl.io/v1alpha1
kind: Device
metadata:
  name: <device_name>
spec:
[...]
  applications:
  - name: wordpress
    1 
    
    image: quay.io/rhem-demos/wordpress-app:latest
    2 
    
    envVars:
    3 
    
      WORDPRESS_DB_HOST: <database_host>
      WORDPRESS_DB_USER: <user_name>
      WORDPRESS_DB_PASSWORD: <password>
[...]
    Copy to Clipboard Toggle word wrap
    1
    A user-defined name for the application that is used when the web console and the CLI list applications.
    2
    A reference to an application package in an OCI registry.
    3
    Optional. A list of key-value pairs that are passed to the deployment tool as environment variables or command line flags.
    Note

    For each application in the applications section of the device specification, you can find the corresponding device status information.

  2. Verify the status of an application deployment on a device by inspecting the device status information by running the following command: 

    flightctl get device/<your_device_id> -o yaml
    Copy to Clipboard Toggle word wrap

    See the following example output: 

    [...]
spec:
  applications:
  - name: example-app
    image: quay.io/flightctl-demos/example-app:v1
status:
  applications:
  - name: example-app
    ready: 3/3
    restarts: 0
    status: Running
  applicationsSummary:
    info: All application workloads are healthy.
    status: Healthy
[...]
    Copy to Clipboard Toggle word wrap

Chapter 8. Device fleets
Copy link

The Red Hat Edge Manager simplifies the management of a large number of devices and workloads through device fleets. A fleet is a resource that defines a group of devices governed by a common device template and management policies.

When you make a change to the device template, all devices in the fleet receive the changes when the Red Hat Edge Manager agent detects the new target specification.

Device monitoring in a fleet is also simplified because you can check the status summary of the whole fleet.

Fleet-level management offers the following advantages:

  • Scales your operations because you perform operations only once for each fleet instead of once for each device.
  • Minimizes the risk of configuration mistakes and configuration drift.

  • Automatically applies the target configuration when you add devices to the fleet or replace devices in the fleet. The fleet specification consists of the following features:

    Label selector
    Determines which devices are part of the fleet.
    Device template
    Defines the configuration that the Red Hat Edge Manager enforces on devices in the fleet.
    Policies
    Governs how devices are managed, for example, how changes to the device template are rolled out to the devices.

You can have both individually managed and fleet-managed devices at the same time.

When you select a device into a fleet, the Red Hat Edge Manager creates the device specification for the new device based on the device template. If you update the device template for a fleet or a new device joins the fleet, the Red Hat Edge Manager enforces the new specification in the fleet.

If a device is not selected into any fleets, the device is considered user-managed or unmanaged. For user-managed devices, you must update the device specification either manually or through an external automation.

Important

A device cannot be a member of more than one fleet at the same time.

For more information, see Labels and label selectors.

8.1. Device selection into a fleet
Copy link

By default, devices are not assigned to a fleet. Instead, each fleet uses a selector that defines which labels a device must have to be added to the fleet.

To understand how to use labels in a fleet, see the following example.

The following list shows point-of-sales terminal devices and their labels:

Expand

Device

Labels

A

type: pos-terminal, region: east, stage: production

B

type: pos-terminal, region: east, stage: development

C

type: pos-terminal, region: west, stage: production

D

type: pos-terminal, region: west, stage: development

If all point-of-sale terminals use the same configuration and are managed by the same operations team, you can define a single fleet called pos-terminals with the type=pos-terminal label selector. Then, the fleet contains devices A, B, C, and D.

However, you might want to create separate fleets for the different organizations for development or production. You can define a fleet for development with the type=pos-terminal, stage=development label selector, which selects devices C and D. Then, you can define another fleet for production with the type=pos-terminal, stage=production label selector. By using the correct label selectors, you can manage both fleets independently.

Important

You must define selectors in a way that two fleets do not select the same device. For example, if one fleet selects region=east, and another fleet selects stage=production, both fleets try to select device A. If two fleets try to select the same device, the Red Hat Edge Manager keeps the device in the currently assigned fleet, if any, and sets the OverlappingSelectors condition on the affected fleets to true.

8.2. Device templates
Copy link

A device template of a fleet contains a device specification that is applied to all devices in the fleet when the template is updated.

For example, you can specify in the device template of a fleet that all devices in the fleet must run the quay.io/flightctl/rhel:9.5 operating system image.

The Red Hat Edge Manager service then rolls out the target specification to all devices in the fleet, and the Red Hat Edge Manager agents update each device.

You can change other specification items in the device template and the Red Hat Edge Manager applies the changes in the same way.

However, sometimes not all of the devices in the fleet need to have the exact same specification. The Red Hat Edge Manager allows templates to contain placeholders that are populated based on the device name or label values.

The syntax of the placeholders matches that of Go templates. However, you can only use simple text and actions.

The use of conditionals or loops in the placeholders is not supported.

You can reference anything from the metadata of a device, such as {{ .metadata.labels.key }} or {{ .metadata.name }}.

You can also use the following functions in your placeholders:

  • The upper function changes the value to uppercase. For example, the function is {{ upper .metadata.name }}.
  • The lower function changes the value to lowercase. For example, the function is {{ lower .metadata.labels.key }}.
  • The replace function replaces all occurrences of a substring with another string. For example, the function is {{ replace "old" "new" .metadata.labels.key }}.
  • The getOrDefault function returns a default value if accessing a missing label. For example, the function is {{ getOrDefault .metadata.labels "key" "default" }}. You can combine the functions in pipelines, for example, a combined function is {{ getOrDefault .metadata.labels "key" "default" | upper | replace " " "-" }}.
Note

Ensure you are using proper Go template syntax. For example, {{ .metadata.labels.target-revision }} is not valid because of the hyphen. Instead, you must refer to the field as {{ index .metadata.labels "target-revision" }}.

You can use the placeholders in device templates in the following ways:

  • You can label devices by deployment stage, for example, stage labels are stage: testing and stage: production. Then, you can use the label with the stage key as placeholder when referencing the operating system image to use, for example, use quay.io/myorg/myimage:latest-{{ .metadata.labels.stage }} or when referencing a configuration folder in a Git repository.
  • You can label devices by deployment site, for example, deployment sites are site: factory-berlin and site: factory-madrid.

  • Then, you can use the label with the site key as parameter when referencing the secret with network access credentials in Kubernetes. The following fields in device templates support placeholders:

    Expand

    Field

    Placeholders supported in

    Operating System Image

    repository name, image name, image tag

    Git Config Provider

    target revision, path

    HTTP Config Provider

    URL suffix, path

    Inline Config Provider

    content, path

8.3. Adding devices to a fleet on the web UI
Copy link

Define the label selector to add devices into a fleet on the web UI.

Complete the following tasks:

Procedure

  1. From the navigation panel, select Application Links → Edge Manager. This opens the external Edge Manager instance.
  2. From the navigation panel, select Fleets. Select the fleet that you want to add devices to.
  3. Click Actions and select Edit fleet.
  4. In the General info tab, click Add label under the Device selector option.
  5. Add the label to select devices for your fleet. Any devices with that label are added to the fleet.

8.4. Adding devices to a fleet on the CLI
Copy link

Define the label selector to add devices into a fleet.

Complete the following tasks:

Procedure

  1. Run the following command to verify that the label selector returns the devices that you want to add to the fleet: 

    flightctl get devices -l type=pos-terminal -l stage=development
    Copy to Clipboard Toggle word wrap

  2. If running the command returns the expected list of devices, you can define a fleet that selects the devices by using the following YAML file: 

    apiVersion: flightctl.io/v1alpha1
kind: Fleet
metadata:
  name: my_fleet
spec:
  selector:
    matchLabels:
      type: pos-terminal
      stage: development
[...]
    Copy to Clipboard Toggle word wrap

  3. Apply the change by running the following command: 

    flightctl apply -f my_fleet.yaml
    Copy to Clipboard Toggle word wrap

  4. Check for any overlaps with the selector of other fleets by running the following command: 

    flightctl get fleets/my_fleet -o json | jq -r '.status.conditions[] | select(.type=="OverlappingSelectors").status'
    Copy to Clipboard Toggle word wrap

    See the following example output: 

    False
    Copy to Clipboard Toggle word wrap

8.5. Rollout device selection
Copy link

When performing a rollout by using flightctl, you must manage which devices participate in the rollout and how much disruption is acceptable. The device selection process and the rollout disruption budget concept ensure controlled and predictable rollouts.

The process and configuration for selecting devices during a rollout includes targeting strategies, batch sequencing, and success criteria for controlled software deployment.

8.5.1. Device targeting
Copy link

A rollout applies only to devices that belong to a fleet. Each device can belong to only a single fleet. Since rollout definitions are done at the fleet level, the selection process determines which devices within a fleet that participate in a batch rollout based on label criteria. After processing all batches, all fleet devices are rolled out.

  • Labels: Devices with specific metadata labels can be targeted for rollouts.
  • Fleet membership: Rollouts apply only to devices within the specified fleet.

8.5.2. Device selection strategy
Copy link

The Red Hat Edge Manager supports only the BatchSequence strategy for device selection. This strategy defines a stepwise rollout process where devices are added in batches based on specific criteria. Batches are executed sequentially. After each batch completes, execution proceeds to the next batch only if the success rate of the previous batch meets or exceeds the configured success threshold.

The success rate is determined as: 

# of successful rollouts in the batch / # of devices in the batch >= success threshold
Copy to Clipboard Toggle word wrap

In a batch sequence, the final batch is an implicit batch and it is not specified in the batch sequence. It selects all devices in a fleet that have not been selected by the explicit batches in the sequence.

8.5.3. Limit in device selection
Copy link

Each batch in the BatchSequence strategy might use an optional limit parameter to define how many devices should be included in the batch. You can specify the limit can in two ways:

  • Absolute number: A fixed number of devices to be selected.

  • Percentage: The percentage of the total matching device population to be selected.

    • If you provide a selector with labels, the percentage is calculated based on the number of devices that match the label criteria within the fleet.
    • If you do not provide a selector, the percentage is applied to all devices in the fleet.

8.5.4. Success threshold
Copy link

The successThreshold defines the percentage of successfully updated devices required to continue the rollout. If the success rate falls below this threshold, the rollout might be paused to prevent further failures.

Example

The following shows an example YAML configuration for a fleet specification: 

apiVersion: v1alpha1
kind: Fleet
metadata:
  name: default
spec:
  selector:
    matchLabels:
      fleet: default
  rolloutPolicy:
    deviceSelection:
      strategy: 'BatchSequence'
      sequence:
        - selector:
            matchLabels:
              site: madrid
          limit: 1  # Absolute number
        - selector:
            matchLabels:
              site: madrid
          limit: 80%  # Percentage of devices matching the label criteria within the fleet
        - limit: 50%  # Percentage of all devices in the fleet
        - selector:
            matchLabels:
              site: paris
        - limit: 80%
        - limit: 100%
    successThreshold: 95%
Copy to Clipboard Toggle word wrap

In this example, there are 6 explicit batches and 1 implicit batch:

  • The first batch selects 1 device having a label site:madrid.
  • With the second batch 80% of all devices having the label site:madrid are either selected for rollout in the current batch or were previously selected for rollout.
  • With the third batch 50% of all devices are either selected for rollout in the current batch or were previously selected for rollout.
  • With the fourth batch all devices that were not previously selected and have the label site:paris are selected.
  • With the fifth batch 80% of all devices are either selected for rollout in the current batch or were previously selected for rollout.
  • With the sixth batch 100% of all devices are either selected for rollout in the current batch or were previously selected for rollout.
  • The last implicit batch selects all devices that have not been selected in any previous batch (might be none).

8.6. Rollout disruption budget
Copy link

A rollout disruption budget defines the acceptable level of service impact during a rollout. This ensures that a deployment does not take down too many devices at once, maintaining overall system stability.

8.6.1. Disruption budget parameters
Copy link

  • groupBy: Defines how devices are grouped when applying the disruption budget. The grouping is done by label keys.
  • minAvailable: Specifies the minimum number of devices that must remain available during a rollout.
  • maxUnavailable: Limits the number of devices that can be unavailable at the same time.

Example

The following shows an example YAML configuration for a fleet specification: 

apiVersion: v1alpha1
kind: Fleet
metadata:
  name: default
spec:
  selector:
    matchLabels:
      fleet: default
  rolloutPolicy:
    disruptionBudget:
      groupBy: ['site', 'function']
      minAvailable: 1
      maxUnavailable: 10
Copy to Clipboard Toggle word wrap

In this example, the grouping is performed on 2 label keys: site and function. A group for disruption budget consists of all devices in a fleet having the same label values for the preceding label keys. For every such group the conditions defined in this specification are continuously enforced.

Chapter 9. Troubleshooting Red Hat Edge Manager
Copy link

When working with devices in Red Hat Edge Manager, you might see issues related to configuration, connectivity, or deployment. Troubleshooting these issues requires understanding how device configurations are applied, how to check logs, and how to verify communication between the device and the service.

The device manifest returned by the flightctl get device command still only has references to external configuration and secret objects. Only when the device agent queries the service, the service replaces the references with the actual configuration and secret data. While this better protects potentially sensitive data, it also makes troubleshooting faulty configurations hard. This is why a user can be authorized to query the effective configuration as rendered by the service to the agent.

Procedure

  • To query the effective configuration, use the following command: 

    flightctl get device/${device_name} --rendered | jq
    Copy to Clipboard Toggle word wrap

9.2. Generating a device log bundle
Copy link

The device includes a script that generates a bundle of logs necessary to debug the agent.

Procedure

  • Run the following command on the device and include the .tar file in the bug report.

    Note

    This depends on an SSH connection to extract the .tar file. 

    sudo flightctl-must-gather
    Copy to Clipboard Toggle word wrap

Legal Notice
Copy link

Copyright © 2025 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat