Chapter 24. Limiting SCHED_OTHER task migration

If a SCHED_OTHER task spawns a large number of other tasks, they will all run on the same CPU. The migration task or softirq will try to balance these tasks so they can run on idle CPUs.

The sched_nr_migrate option can be adjusted to specify the number of tasks that will move at a time. Because real-time tasks have a different way to migrate, they are not directly affected by this. However, when softirq moves the tasks, it locks the run queue spinlock, thus disabling interrupts.

If there are a large number of tasks that need to be moved, it occurs while interrupts are disabled, so no timer events or wakeups will be allowed to happen simultaneously. This can cause severe latencies for real-time tasks when sched_nr_migrate is set to a large value.

Increasing the sched_nr_migrate variable provides high performance from SCHED_OTHER threads that spawn many tasks at the expense of real-time latency.

For low real-time task latency at the expense of SCHED_OTHER task performance, the value must be lowered. The default value is 8.