MEDIUM: task: add a new flag TASK_RT to permit a task to skip the priority queue

For some very rare tasks that need to be woken up at an exact date (right
now the only known use case is haload's periodic stats collection), it's
currently difficult to guarantee the wake up date on a heavily loaded
run queue.

This patch introduces TASK_RT for real-time tasks. Right now, all it does
is modify __task_wakeup() to immediately switch to __tasklet_wakeup_*()
and effectively bypass the priority-based run queue. Doing it here has
the benefit of making sure that it automatically applies to tasks found
in the wait queue, and that it will also work for _task_drop_running().

For now nothing uses it. The doc was updated.

doc/internals/api/scheduler.txt

@@ -272,6 +272,18 @@ application, both for tasks and tasklets:
                        sense to reset this flag from the ->process() function
                        itself.
 
+  - TASK_RT            when set, indicates that the task has real-time
+                       constraints (this is ignored for tasklets). Such a task
+                       will bypass the priority ordering of the run queue in
+                       order to guarantee a wakeup time as close as possible to
+                       the scheduled one even under load. Only one such task
+                       may be executed per round so this must be restricted to
+                       a few timing-critical tasks only (those for which a one
+                       millisecond skew is not acceptable). Such tasks are not
+                       meant to be woken up by other threads than the one they
+                       are supposed to run on, otherwise(their constraints may
+                       not be honored.
+
   - TASK_HEAVY         when set, indicates that this task does so heavy
                        processing that it will become mandatory to give back
                        control to I/Os otherwise big latencies might occur. It

include/haproxy/task-t.h

@@ -37,7 +37,7 @@
 #define TASK_RUNNING      0x00000001  /* the task is currently running */
 /* unused                 0x00000002 */
 #define TASK_QUEUED       0x00000004  /* The task has been (re-)added to the run queue */
-/* unused                 0x00000008 */
+#define TASK_RT           0x00000008  /* This task always skips the priority queue */
 #define TASK_SELF_WAKING  0x00000010  /* task/tasklet found waking itself */
 #define TASK_KILLED       0x00000020  /* task/tasklet killed, may now be freed */
 #define TASK_HEAVY        0x00000080  /* this task/tasklet is extremely heavy */
@@ -65,7 +65,7 @@
 /* unused: 0x400000..0x80000000 */
 
 /* These flags are persistent across scheduler calls */
-#define TASK_PERSISTENT   (TASK_SELF_WAKING | TASK_KILLED | \
+#define TASK_PERSISTENT   (TASK_SELF_WAKING | TASK_KILLED | TASK_RT | \
                            TASK_HEAVY | TASK_F_TASKLET | TASK_F_USR1 | \
                            TASK_F_WANTS_TIME)
 

src/task.c

@@ -242,6 +242,15 @@ void __task_wakeup(struct task *t)
 
 	BUG_ON(t->tid == -1);
 
+	if (unlikely(_HA_ATOMIC_LOAD(&t->state) & TASK_RT)) {
+		/* real-time tasks must be super rare; they are woken up as tasklets. */
+		if (thr < 0 || thr == tid)
+			__tasklet_wakeup_here((struct tasklet *)t);
+		else
+			__tasklet_wakeup_on((struct tasklet *)t, thr);
+		return;
+	}
+
 #ifdef USE_THREAD
 	if (thr != tid) {
 		root = &ha_thread_ctx[thr].rqueue_shared;