Scheduler flag to enable dispatch from scheduler thread *with* pending events

[gparmer]

Background

The kernel maintains a list of asynchronous events to be delivered to the (core's) scheduler thread. These include the activation of threads through asnds from interrupts, the preemption of threads when those asnds activate a thread, and the consumption of time through execution of interrupt threads. In short, all of the events that bypass the scheduler (asynchronous activations) are tracked this way.

To compensate for race conditions, if a dispatch is ever attempted, and the scheduler thread has pending events, we switch to the scheduler thread so that it can clear them out, thus making a more accurate scheduling decision. However, an annoying edge case is this: If the scheduler thread knows that there are events to process, and tries to take the scheduler critical section, and another thread is already in the critical section, then the scheduler thread cannot switch to that thread (given the logic from the previous paragraph). Thus, we're stuck: we cannot process the events as we cannot take the critical section.

The current code simply loops removing all scheduler events adding them to a linked list, then once they are all processed, it will attempt to take the critical section, and apply the side effects of the events (i.e. wake/block threads).

Problems

The problems with the above approach:

1. We wish to maintain TCap inheritance during scheduler thread execution. But we cannot track that when the events aren't pending. This is a "light" requirement, as using only the core TCap's priority should be correct as it should have highest priority.
2. Requiring the doubly linked list in the scheduler core is going to make verification very hard.
3. We are delaying applying the effects of the events.

The big upside of this approach is that the event retrieval is completely outside of the critical section. From a worst-case perspective, I don't think this matters much as a high-priority thread has to wait for the CS regardless.

Proposal

Add a flag to thread dispatch that only the scheduler thread can pass that enables a dispatch even if there are pending scheduler events. This flag should only be used in the case that the scheduler thread is switching to a thread that has the scheduler CS, thus it will be re-activated to process the events ASAP. This removes the linked list at user-level, and simplifies the scheduler loop while adding very little complication to the kernel.