SupportConsoleDevelopersStart a trialContact

Chapter 5. Using jobs and daemon sets

Daemon sets ensure that pods run on all or specific nodes in a cluster, enabling cluster-wide services such as logging agents, monitoring tools, and shared storage that must be present on every node.

As an administrator, you can create and use daemon sets to run replicas of a pod on specific or all nodes in a Red Hat OpenShift Service on AWS classic architecture cluster.

A daemon set ensures that all (or some) nodes run a copy of a pod. As nodes are added to the cluster, pods are added to the cluster. As nodes are removed from the cluster, those pods are removed through garbage collection. Deleting a daemon set cleans up the pods it created.

You can use daemon sets to create shared storage, run a logging pod on every node in your cluster, or deploy a monitoring agent on every node.

For security reasons, the cluster administrators and the project administrators can create daemon sets.

Important

Daemon set scheduling is incompatible with the project’s default node selector. If you fail to disable it, the daemon set gets restricted by merging with the default node selector. This results in frequent pod recreates on the nodes that got unselected by the merged node selector, which in turn puts unwanted load on the cluster.

5.1.1. Daemon set scheduling
Copy link

A daemon set ensures that all eligible nodes run a copy of a pod by using the default scheduler, which replaces the original daemon set controller scheduling approach for improved consistency and preemption support.

Normally, the node that a pod runs on is selected by the Kubernetes scheduler. However, daemon set pods are created and scheduled by the daemon set controller. That introduces the following issues:

  • Inconsistent pod behavior: Normal pods waiting to be scheduled are created and in Pending state, but daemon set pods are not created in Pending state. This is confusing to the user.
  • Pod preemption is handled by default scheduler. When preemption is enabled, the daemon set controller makes scheduling decisions without considering pod priority and preemption.

The ScheduleDaemonSetPods feature, enabled by default in Red Hat OpenShift Service on AWS classic architecture, lets you schedule daemon sets using the default scheduler instead of the daemon set controller, by adding the nodeAffinity term to the daemon set pods, instead of the spec.nodeName term. The default scheduler is then used to bind the pod to the target host. If node affinity of the daemon set pod already exists, it is replaced. The daemon set controller only performs these operations when creating or modifying daemon set pods, and no changes are made to the spec.template of the daemon set.

Example nodeAffinity in a daemon set pod

kind: Pod
apiVersion: v1
metadata:
  name: hello-node-6fbccf8d9-9tmzr
#...
spec:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchFields:
        - key: metadata.name
          operator: In
          values:
          - target-host-name
#...

In addition, a node.kubernetes.io/unschedulable:NoSchedule toleration is added automatically to daemon set pods. The default scheduler ignores unschedulable Nodes when scheduling daemon set pods.

5.1.2. Creating daemonsets
Copy link

Create a daemon set to automatically deploy and maintain pod replicas across selected nodes in your cluster, using node selectors to control placement.

When creating daemon sets, the nodeSelector field is used to indicate the nodes on which the daemon set should deploy replicas.

Important
  • If you update a daemon set pod template, the existing pod replicas are not affected.
  • If you delete a daemon set and then create a new daemon set with a different template but the same label selector, it recognizes any existing pod replicas as having matching labels. It does not update them or create new replicas despite a mismatch in the pod template.
  • If you change node labels, the daemon set adds pods to nodes that match the new labels and deletes pods from nodes that do not match the new labels.

To update a daemon set, force new pod replicas to be created by deleting the old replicas or nodes.

Prerequisites

  • The default project-wide node selector in your namespace is set to an empty string by using the openshift.io/node-selector annotation: 

    $ oc patch namespace myproject -p \
    '{"metadata": {"annotations": {"openshift.io/node-selector": ""}}}'
    Tip

    You can alternatively apply the following YAML to disable the default project-wide node selector for a namespace: 

    apiVersion: v1
kind: Namespace
metadata:
  name: <namespace>
  annotations:
    openshift.io/node-selector: ''
#...

Procedure

  1. Define the daemon set yaml file: 

    apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: hello-daemonset
spec:
  selector:
      matchLabels:
        name: hello-daemonset
  template:
    metadata:
      labels:
        name: hello-daemonset
    spec:
      nodeSelector:
        role: worker
      containers:
      - image: openshift/hello-openshift
        imagePullPolicy: Always
        name: registry
        ports:
        - containerPort: 80
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
      serviceAccount: default
      terminationGracePeriodSeconds: 10
#...

    where:

    spec.selector.matchLabels.name
    Specifies the label selector that determines which pods belong to the daemon set.
    spec.template.metadata.labels.name
    Specifies the pod template’s label selector. Must match the label selector above.
    spec.template.spec.nodeSelector
    Specifies the node selector that determines on which nodes pod replicas should be deployed. A matching label must be present on the node.

  2. Create the daemon set object by running the following command: 

    $ oc create -f daemonset.yaml

Verification

  • Verify that the pods were created and that each node has a pod replica by running the following command: 

    $ oc get pods

    Example output

    hello-daemonset-cx6md   1/1       Running   0          2m
hello-daemonset-e3md9   1/1       Running   0          2m

  • Verify that the pods are placed onto the correct nodes by running the following command: 

    $ oc describe pod/hello-daemonset-cx6md|grep Node

    Example output

    Node:        openshift-node01.hostname.com/10.14.20.134

5.2. Running tasks in pods using jobs
Copy link

Jobs execute one-time or scheduled tasks in your cluster, tracking completion status and managing pod lifecycles for batch workloads, parallel processing, and recurring operations.

A job executes a task in your Red Hat OpenShift Service on AWS classic architecture cluster.

A job tracks the overall progress of a task and updates its status with information about active, succeeded, and failed pods. Deleting a job cleans up any pod replicas it created. Jobs are part of the Kubernetes API, which can be managed with oc commands like other object types.

Sample Job specification

apiVersion: batch/v1
kind: Job
metadata:
  name: pi
spec:
  parallelism: 1
  completions: 1
  activeDeadlineSeconds: 1800
  backoffLimit: 6
  ttlSecondsAfterFinished: 100
  template:
    metadata:
      name: pi
    spec:
      containers:
      - name: pi
        image: perl
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: OnFailure
#...

where:

spec.parallelism
Specifies the pod replicas a job should run in parallel.
spec.completions
Specifies the successful pod completions needed to mark a job completed.
spec.activeDeadlineSeconds
Specifies the maximum duration the job can run.
spec.backoffLimit
Specifies the number of retries for a job.
spec.ttlSecondsAfterFinished
Specifies the period of time in seconds after which the job should be automatically deleted upon completion.
spec.template
Specifies the template for the pod the controller creates.
spec.template.spec.restartPolicy
Specifies the restart policy of the pod.

Additional resources

5.2.1. Understanding jobs and cron jobs
Copy link

Jobs run one-time tasks and track completion status, while cron jobs schedule recurring tasks using cron expressions, supporting non-parallel, parallel with fixed completion count, and work queue execution patterns.

A job tracks the overall progress of a task and updates its status with information about active, succeeded, and failed pods. Deleting a job cleans up any pods it created. Jobs are part of the Kubernetes API, which can be managed with oc commands like other object types.

There are two possible resource types that allow creating run-once objects in Red Hat OpenShift Service on AWS classic architecture:

Job

A regular job is a run-once object that creates a task and ensures the job finishes.

There are three main types of task suitable to run as a job:

  • Non-parallel jobs:

    • A job that starts only one pod, unless the pod fails.
    • The job is complete as soon as its pod terminates successfully.

  • Parallel jobs with a fixed completion count:

    • a job that starts multiple pods.
    • The job represents the overall task and is complete when there is one successful pod for each value in the range 1 to the completions value.

  • Parallel jobs with a work queue:

    • A job with multiple parallel worker processes in a given pod.
    • Red Hat OpenShift Service on AWS classic architecture coordinates pods to determine what each should work on or use an external queue service.
    • Each pod is independently capable of determining whether all peer pods are complete and that the entire job is done.
    • When any pod from the job terminates with success, no new pods are created.
    • When at least one pod has terminated with success and all pods are terminated, the job is successfully completed.
    • When any pod has exited with success, no other pod should be doing any work for this task or writing any output. Pods should all be in the process of exiting.
Cron job

A job can be scheduled to run multiple times, using a cron job.

You can use a cron job to specify how the job runs. Cron jobs are part of the Kubernetes API, which can be managed with oc commands like other object types.

Cron jobs are useful for creating periodic and recurring tasks, such as running backups or sending emails. Cron jobs can also schedule individual tasks for a specific time, such as if you want to schedule a job for a low activity period. A cron job creates a Job object based on the time zone configured on the control plane node that runs the cronjob controller.

Warning

A cron job creates a Job object approximately once per execution time of its schedule, but there are circumstances in which it fails to create a job or two jobs might be created. Therefore, jobs must be idempotent and you must configure history limits.

5.2.1.1. Understanding how to create jobs
Copy link

Both resource types require a job configuration that consists of the following key parts:

  • A pod template, which describes the pod that Red Hat OpenShift Service on AWS classic architecture creates.

  • The parallelism parameter, which specifies how many pods running in parallel at any point in time should execute a job.

    • For non-parallel jobs, leave unset. When unset, defaults to 1.

  • The completions parameter, specifying how many successful pod completions are needed to finish a job.

    • For non-parallel jobs, leave unset. When unset, defaults to 1.
    • For parallel jobs with a fixed completion count, specify a value.
    • For parallel jobs with a work queue, leave unset. When unset defaults to the parallelism value.

When defining a job, you can define its maximum duration by setting the activeDeadlineSeconds field. It is specified in seconds and is not set by default. When not set, there is no maximum duration enforced.

The maximum duration is counted from the time when a first pod gets scheduled in the system, and defines how long a job can be active. It tracks overall time of an execution. After reaching the specified timeout, the job is terminated by Red Hat OpenShift Service on AWS classic architecture.

A job can be considered failed, after a set amount of retries due to a logical error in configuration or other similar reasons. Failed pods associated with the job are recreated by the controller with an exponential back off delay (10s, 20s, 40s …) capped at six minutes. The limit is reset if no new failed pods appear between controller checks.

Use the spec.backoffLimit parameter to set the number of retries for a job.

Cron jobs can leave behind artifact resources such as jobs or pods. As a user it is important to configure history limits so that old jobs and their pods are properly cleaned. There are two fields within cron job’s spec responsible for that:

  • .spec.successfulJobsHistoryLimit. The number of successful finished jobs to retain (defaults to 3).
  • .spec.failedJobsHistoryLimit. The number of failed finished jobs to retain (defaults to 1).

You can specify that Red Hat OpenShift Service on AWS classic architecture should delete jobs when they are finished, either Complete or Failed, in order to limit the number of objects in your cluster.

By default, for jobs that are not associated with a cron job or other controlling object, Red Hat OpenShift Service on AWS classic architecture does not delete the job or its related pods when the job is finished. If you keep these artifacts in your cluster, you can view the status of finished jobs or view job logs for errors, warnings, or other diagnostic output.

However, having too many of these objects in the cluster can lead to etcd issues, latency issues, performance degradation, and other problems.

You can configure a time-to-live (TTL) duration for jobs without a controlling object by setting the ttlSecondsAfterFinished parameter in the job spec. When set, Red Hat OpenShift Service on AWS classic architecture deletes the jobs and any related pods immediately after the job is finished or after a specified period of time in seconds.

5.2.1.6. Known limitations
Copy link

The job specification restart policy only applies to the pods, and not the job controller. However, the job controller is hard-coded to keep retrying jobs to completion.

As such, restartPolicy: Never or --restart=Never results in the same behavior as restartPolicy: OnFailure or --restart=OnFailure. That is, when a job fails it is restarted automatically until it succeeds (or is manually discarded). The policy only sets which subsystem performs the restart.

With the Never policy, the job controller performs the restart. With each attempt, the job controller increments the number of failures in the job status and creates new pods. This means that with each failed attempt, the number of pods increases.

With the OnFailure policy, kubelet performs the restart. Each attempt does not increment the number of failures in the job status. In addition, kubelet retries failed jobs starting pods on the same nodes.

5.2.2. Creating jobs
Copy link

Create a job to run a one-time task, configuring parallelism, completion count, time limits, retry policies, and automatic cleanup behavior.

You create a job in Red Hat OpenShift Service on AWS classic architecture by creating a job object.

Note

You can also create and launch a job from a single command using oc create job. The following command creates and launches a job similar to the one specified in the following example: 

$ oc create job pi --image=perl -- perl -Mbignum=bpi -wle 'print bpi(2000)'

Procedure

  1. Create a YAML file similar to the following: 

    apiVersion: batch/v1
kind: Job
metadata:
  name: pi
spec:
  parallelism: 1
  completions: 1
  activeDeadlineSeconds: 1800
  backoffLimit: 6
  ttlSecondsAfterFinished: 100
  template:
    metadata:
      name: pi
    spec:
      containers:
      - name: pi
        image: perl
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: OnFailure
#...

    where:

    spec.parallelism
    Specifies how many pod replicas a job should run in parallel; defaults to 1. For non-parallel jobs, leave unset. When unset, defaults to 1. This value is optional.
    spec.completions
    Specifies how many successful pod completions are needed to mark a job completed. For non-parallel jobs, leave unset. When unset, defaults to 1. For parallel jobs with a fixed completion count, specify the number of completions. For parallel jobs with a work queue, leave unset. When unset defaults to the parallelism value. This value is optional.
    spec.activeDeadlineSeconds
    Specifies the maximum duration the job can run. This value is optional.
    spec.backoffLimit
    Specifies the number of retries for a job. This field defaults to six. This value is optional.
    spec.ttlSecondsAfterFinished
    Specifies the period of time in seconds after which the job should be automatically deleted upon completion. If set to '0', the job is immediately deleted after completion. If this field is not included, the job is not automatically deleted. This value is optional.
    spec.template
    Specifies the template for the pod the controller creates.
    spec.template.spec.restartPolicy
    Specifies the restart policy of the pod: Never (Do not restart the job), OnFailure (Restart the job only if it fails), or Always (Always restart the job). For details on how Red Hat OpenShift Service on AWS classic architecture uses restart policy with failed containers, see the Example States in the Kubernetes documentation.

  2. Create the job: 

    $ oc create -f <file-name>.yaml

5.2.3. Creating cron jobs
Copy link

Create a cron job to schedule recurring tasks using cron expressions, configuring time zone, concurrency policies, history limits, and suspension behavior.

You create a cron job in Red Hat OpenShift Service on AWS classic architecture by creating a job object.

Note

You can also create and launch a cron job from a single command using oc create cronjob. The following command creates and launches a cron job similar to the one specified in the following example: 

$ oc create cronjob pi --image=perl --schedule='*/1 * * * *' -- perl -Mbignum=bpi -wle 'print bpi(2000)'

With oc create cronjob, the --schedule option accepts schedules in cron format.

Procedure

  1. Create a YAML file similar to the following: 

    apiVersion: batch/v1
kind: CronJob
metadata:
  name: pi
spec:
  schedule: "*/1 * * * *"
  concurrencyPolicy: "Replace"
  startingDeadlineSeconds: 200
  suspend: true
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            parent: "cronjobpi"
        spec:
          containers:
          - name: pi
            image: perl
            command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
          restartPolicy: OnFailure

    where:

    spec.schedule
    Specifies the schedule for the job in cron format. In this example, the job will run every minute.
    spec.concurrencyPolicy
    Specifies how to treat concurrent jobs within a cron job. Only one of the following concurrent policies may be specified. If not specified, this defaults to allowing concurrent executions: Allow (allows cron jobs to run concurrently), Forbid (forbids concurrent runs, skipping the next run if the previous has not finished yet), or Replace (cancels the currently running job and replaces it with a new one). This value is optional.
    spec.startingDeadlineSeconds
    Specifies a deadline (in seconds) for starting the job if it misses its scheduled time for any reason. Missed jobs executions will be counted as failed ones. If not specified, there is no deadline. This value is optional.
    spec.suspend
    Specifies a flag allowing the suspension of a cron job. If set to true, all subsequent executions will be suspended. This value is optional.
    spec.successfulJobsHistoryLimit
    Specifies the number of successful finished jobs to retain (defaults to 3).
    spec.failedJobsHistoryLimit
    Specifies the number of failed finished jobs to retain (defaults to 1).
    spec.jobTemplate
    Specifies the job template. This is similar to the job example.
    spec.jobTemplate.spec.template.metadata.labels
    Specifies labels for jobs spawned by this cron job.
    spec.jobTemplate.spec.template.spec.restartPolicy

    Specifies the restart policy of the pod. This does not apply to the job controller.

    The .spec.successfulJobsHistoryLimit and .spec.failedJobsHistoryLimit fields are optional. These fields specify how many completed and failed jobs should be kept. By default, they are set to 3 and 1 respectively. Setting a limit to 0 corresponds to keeping none of the corresponding kind of jobs after they finish.

  2. Create the cron job: 

    $ oc create -f <file-name>.yaml
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top