sched/core: Call wakeup_preempt() for queued tasks during class change

In the Linux kernel, the following vulnerability has been resolved:

sched/core: Call wakeup_preempt() for queued tasks during class change

Paul reported recent RCU changes exposed a lost resched where CPU idles with a rcutorture thread waiting while being completely runnable.

Dumping the rq state at the time of idling with runnable tasks showed the following:

rq->next_class = &fair_sched_class rq->nr_running = 1 rq->cfs.h_nr_queued = 0 rq->rt.rt_nr_running = 1

Although the waiting task was an RT task, "rq->next_class" did not point to rt_sched_class and was stuck at the lower priority fair_sched_class.

Inspecting sched_change_end() revealed the following race with __sched_setscheduler() for queued, but non-running tasks:

CPU0                                   CPU1
====                                   ====

__schedule()

/*
 * rq->donor is a fair task that blocks.
 * rq->next_class = &fair_sched_class
 */

rq_lock(rq0)

pick_next_task()
  /* nr_running = 0 */
  pick_task_fair()
    sched_balance_newidle()
      raw_spin_rq_unlock(rq0)
                                rq_lock(rq0)

                                activate_task(rq0, p)
                                  /* p is a fair task */
                                  rq0->nr_running++;
                                  rq0->cfs.h_nr_queued++;
                                  wakeup_preempt(rq, p)
                                    rq->next_class = &fair_sched_class
                                    resched_curr()

                                rq_unlock(rq0)

                                __sched_setscheduler(p)
                                  /* p is moved to SCHED_RR */
                                  task_rq_lock(p)
                                  sched_change_begin()

			      rq0->cfs.h_nr_queued--;
			      rq0->rt.rt_nr_running++;

                                  sched_change_end()
                                    /*
                                     * p->on_rq = 1 (ctx->queued)
                                     * but p is not rq->donor
                                     * (!ctx->running) hence
                                     * resched_curr() is and
                                     * rq->next_class upgrade is
                                     * skipped.
                                     */

                                   task_rq_unlock()

      raw_spin_rq_lock(rq0)

      /*
       * rq->nr_running = 1
       * rq->cfs.h_nr_queued = 0
       * rq->rt.rt_nr_running = 1
       * rq->next_class = &fair_sched_class
       */

       if (rq->cfs.h_nr_queued && ...)
         /* Above is false */

       if (rq_modified_above(this_rq, &fair_sched_class))
         /* Also false since rq->next_class is still fair */

  /*
   * sched_balance_newidle() is considered to
   * have failed to pull any fair tasks and no
   * other wakeups were detected.
   */
   return NULL; /* pick_task_fair() */

pick_task_idle()
  return rq->idle;

next = rq->idle;

/* Preemption indicators are lost */
clear_tsk_need_resched(prev);
clear_preempt_need_resched();

rq->curr = rq->idle;

!!! rq->idle switches in with rq->running == 1 but !need_resched() !!!

In simple words, enqueue, followed by a class change of the newly queued task goes unnoticed in rq->next_class indicator and sched_balance_newidle() sees no fair task, nor any higher priority task enqueued.

The above sequence will lead to CPU idling with a runnable task on the CPU.

Call wakeup_preempt() for ctx->queued cases in sched_change_end() to re-evaluate rq->next_class after class change (ENQUEUE_CLASS) and set the NEED_RESCHED indicator accordingly.

Newidle balance will correctly see an enqueue of a higher priority t ---truncated---