Concern: flow control

[matejpavlovic]

Original concern

Mir's architecture doesn't seem to offer any
convenient mechanism to enable flow control within the system.
Conversely, it seems to demand non-blocking operation for most
event types as well as implicitly rely on unbounded buffers to hold
events waiting for being processed. One can argue that there is
logically an upper bound on the amount of internally-produced
events waiting in the event buffers. However, when it comes to
external events, such as reception of messages from potentially
malicious peer nodes, or potentially blocking or time-consuming
operations, such as appending to a persistent write-ahead log or
processing of ordered requests by the application, there must be
some kind of flow-control mechanism. The event-based architecture
of Mir seems to require implementing some non-trivial inter-module
flow-control protocols in such cases.

(copied @sergefdrv 's post in umbrella discussion #46 )

[matejpavlovic]

Personally, I do not think that unbounded buffers are an issue in this case. I base my reasoning on the following assumption:

• Regardless of the initial state of the internal event buffers,
  if the node is "left alone" (i.e. stops being triggered externally),
  the buffers will eventually all become empty.

It can happen that the size of those buffers temporarily grows before that,
but, as you said, there is a logical limit to how much they grow and I do not think there is a reason for them to grow significantly.
I think that any reasonable implementation satisfies that assumption.
Otherwise it would mean that there is an infinite positive feedback loop in the implementation, which I can only see as a bug.

The only sources of externally triggered events are the network (producing MessageReceived events)
and the timer (producing Tick/Timeout events).
If necessary, we can rather trivially enforce flow control by simply not processing those events when the event buffers are full.
Concretely, if the size of the event buffers exceeds some threshold,
we can stop reading events from the Net and Timer modules' channels until the buffer size decreases again.
Ultimately, those modules will stop receiving messages from the network / generating timeouts for as long as needed.

It is true that some modules might be slower at processing events than others,
but this should not have an impact on the unboundedness of buffers.

[sergefdrv]

@matejpavlovic There seem to be a contradiction. You state that (1) events never get lost, (2) each module has its own buffer for incoming events and its own thread(s) of execution, (3) the dispatcher thread does not depend any module's execution speed. If there is a slow module that cannot process incoming events in time then its incoming event buffer must grow indefinitely. Where is the flow control then?

[matejpavlovic]

Sorry, I used a too vague expression that the dispatcher "does not depend on any module's execution speed".
If a slow module cannot process incoming events in time, its buffer will still not grow indefinitely, because there will not be an infinite number of events to write in that buffer. When a buffer grows too big, we stop the inflow of "external" events (messages and time events) to the system (not yet implemented, but described above and very easy to implement). In this sense, the dispatcher does depend on the module's execution speed. What I meant is that it can never get stuck or otherwise fail just because of a slow module, sorry for the wrong expression. The principle of its operation also does not change, we just make two channels unavailable to it if necessary.

[matejpavlovic]

In the very extreme case (might be quite useful for debug mode), we can even block incoming messages and timer events until all buffers are empty. This would reduce the required buffer sizes to an absolute minimum, but prevent efficient pipelining of event processing. Intuitively, however, I expect that even very small buffers will enable leveraging most of the pipelining and concurrency potential.

[sergefdrv]

@matejpavlovic Do I understand correctly that you consider handing external events differently than internal ones, i.e. stop processing external events if internal event buffers become full?

[matejpavlovic]

Yes, that's the idea.