scx: Exit a scheduler for unhandled hotplug events #177
Conversation
We currently provide scx_ops_error() as a way for ext.c to cause a scheduler to be evicted due to erroneous behavior (for example, due to returning an invalid CPU from ops.select_cpu()). Now that we have a method for exiting gracefully with an exit code from BPF programs, we can similarly provide an scx_ops_exit() macro that allows ext.c to exit and pipe an exit code up to user space. This patch adds that macro. A subsequent patch will use it to exit and plumb up hotplug events. Signed-off-by: David Vernet <[email protected]>
Byte-Lab
force-pushed
the
hotplug_restart
branch
from
eb03eb3
to
ef91247
Compare
A scheduler may implement ops.cpu_online() and ops.cpu_offline() for CPU onlining and offlining events respectively. If a scheduler does _not_ implement these callbacks, it's indicative that they do not support hotplug. Given that a scheduler that doesn't support hotplug is essentially guaranteed to perform incorrectly if a hotplug event occurs, let's update ext.c to do the sane thing, and exit the scheduler automatically. Given that there may be other events in the future that could cause the scheduler to initiate an exit, we create a new enum scx_exit_code type that reserves the top bit of the exit_code field in struct scx_exit_info, and defines SCX_ECODE_RESTART. We could as an alternative instead just return something like -EAGAIN to signify that user space can try restarting the scheduler. Signed-off-by: David Vernet <[email protected]>
Now that we have bits reserved for system exit code reasons and actions, as well as bits available for user by user space, let's add some ease-of-use macro to user_exit_info.h. A subsequent patch will add selftests that use these macros. Signed-off-by: David Vernet <[email protected]>
We've recently added some logic related to hotplug: - If a hotplug event occurs and a scheduler hasn't implemented a callback for it, we automatically exit the scheduler with specific, built-in exit codes - With scx_bpf_exit(), a scheduler can choose to manually exit the scheduler in a hotplug event, or do something else. In any case, the scheduler should _not_ be automatically exited by the kernel Let's add selftests to validate these conditions. Signed-off-by: David Vernet <[email protected]>
Byte-Lab
force-pushed
the
hotplug_restart
branch
from
ef91247
to
e77fa31
Compare
|
I'm not sure if relying purely on the implementation of |
That would indeed be more convenient for I'd expect that the cases where the scheduler cares about hotplug are when they have some view of the host topology; and generally speaking I'd expect that to be the norm for most production schedulers. I think the question is whether we want to do what we think is "the right thing" in terms of scheduler correctness for the common case (meaning assuming a scheduler that doesn't implement hotplug is probably not going to do the right thing in the event of a hotplug event), or to instead have a standalone flag that completely controls the behavior. Given that I think it's probably atypical for a scheduler to be completely agnostic to hotplug changes, my inclination is to err on the side of always exiting the scheduler if the callbacks aren't defined. Wdyt? |
|
Hm... good point about implementing an empty That said, I agree with this approach, thanks for clarifying it. |
A scheduler may implement ops.cpu_online() and ops.cpu_offline() for CPU
onlining and offlining events respectively. If a scheduler does not
implement these callbacks, it's indicative that they do not support
hotplug. Given that a scheduler that doesn't support hotplug is
essentially guaranteed to perform incorrectly if a hotplug event occurs,
let's update ext.c to do the sane thing, and exit the scheduler
automatically.
Given that there may be other events in the future that could cause the
scheduler to initiate an exit, we create a new enum scx_exit_code type
that reserves the top bit of the exit_code field in struct
scx_exit_info, and defines SCX_ECODE_RESTART. We could as an alternative
instead just return something like -EAGAIN to signify that user space
can try restarting the scheduler.
Note that this isn't a 100% foolproof, race free hotplug detection mechanism.
For some schedulers, if a CPU is hotplugged after inspecting the host
topology but before attaching the scheduler, we could run into problems.
We'll need something like a hotplug generation counter to accommodate
this. We can take care of it in a separate follow up patch set.