Error handling

The idea with actor systems is that you fail fast and recover quickly. If your state value (that you returned from Setup) implements IDisposable it will be cleaned up automatically.

By default messages that cause errors are redirected to the DeadLetters process. You can subscribe to the DeadLetters to see all undelivered messages:

   Process.observe(Process.DeadLetters).Subscribe(Console.WriteLine);

For different strategies to failure (backing off, redirection of messages, etc.) use the strategy system. The way it works is this:

• The parent process has a strategy to deal with failure of its children
• Each parent process runs a State<StrategyContext, Unit> monad
• The monad manages state over the time for successful and unsuccessful message processing
  • For example it holds the number of failures, when the last failure was, etc.
• The strategy is essentially a mini-DSL that has two main outputs:
  • Directive
  • MessageDirective
• Directive is the instruction to the parent of the failed child process on what to do when its child fails. It can be:
  • Resume meaning ignore the error, don't reset the child process, and carry on with the same state.
  • Stop meaning shutdown the child process (disposing of the state if necessary)
  • Restart (the default) meaning dispose the state object, and restart the process by calling its setup function and then moving on to the next message.
  • Escalate means that the parent of the failed child can't deal with the error, so push it up one level to the grand-parent.
• MessageDirective is the instruction on what to do with the message that caused the failure. It can be:
  • ForwardToDeadLetters (the default) - Sends the failed message to the DeadLetters system process
  • ForwardToSelf - Puts the message on the back of the queue to be processed again
  • ForwardToParent - The parent of the failed child will get the message
  • ForwardTo(pid) - Send to a specific process for handling
  • StayInQueue - Means remain at the front of the queue for when the process has recovered (could cause blocking for permanent failures)
• There is also a concept of AllForOne and OneForOne. Because its the parent of the failed process that runs the strategy, it can also force all child processes to respond to the strategy. So for example if you had a download process, which had 10 child processed all downloading parts of the final piece, and one failed, then you could use AllForOne to force all child processes to restart, stop, or whatever.

Here's a code example of a strategy:

        State<StrategyContext, Unit> retriesAndBackOff =
            OneForOne(
                Retries(Count: 5, Duration: 30*seconds),
                Backoff(Min: 2*seconds, Max: 1*hour, Step: 5*seconds),
                Match(
                    With<NotImplementedException>(Directive.Stop),
                    With<ArgumentNullException>(Directive.Escalate),
                    Otherwise(Directive.Restart)),
                Redirect(
                    When<Restart>(MessageDirective.ForwardToParent),
                    When<Escalate>(MessageDirective.ForwardToSelf),
                    Otherwise(MessageDirective.ForwardToDeadLetters)));

That affects only the process failing. If there are 5 retries in 30 seconds then the process will shut-down. There is also a back-off strategy that starts with a 2 second back off, and each failure will add 5 seconds to the delay. If at any point a message is processed successfully then the back-off time is reset. However if it backs off to the point of there being a 1 hour delay then the process shuts down.

• The Match part allows you to match the type of exception to have a different strategy.
• The Redirect section matches the Directive and makes a decision what to do with the message.

If you always want to restart and forward to deadletters, then there's a shorthand:

        State<StrategyContext, Unit> retriesAndBackOff =
            OneForOne(
                Retries(Count: 5, Duration: 30*seconds),
                Backoff(Min: 2*seconds, Max: 1*hour, Step: 5*seconds),
                Always(Directive.Restart),
                Redirect(MessageDirective.ForwardToDeadLetters));

But actually they're the default anyway, so you can do this:

        State<StrategyContext, Unit> retriesAndBackOff =
            OneForOne(
                Retries(Count: 5, Duration: 30*seconds),
                Backoff(Min: 2*seconds, Max: 1*hour, Step: 5*seconds));

It's possible to build your own strategies as long as they're state monads. Just take a look at the existing ones for guidance.

On top of this there's a full DSL in the conf file system, which makes authoring these strategies much easier:

let   default-retries:  3
let   strSetting:       "testing 123"
float dblSetting:       1.25

strategy testStrat1: 
    one-for-one:
        retries: count = default-retries + 2, duration = 10 seconds
        backoff: 1 seconds

        always: restart

        redirect when
         | restart  -> forward-to-self
         | stop     -> forward-to-dead-letters

strategy testStrat2: 
    all-for-one:
        retries: 5
        backoff: min = 1 seconds, max = 100 seconds, step = 2 second

        match
         | LanguageExt.ProcessSetupException -> restart

        redirect when
         | restart  -> forward-to-self

process test1Supervisor:
        pid:          /root/user/test1-supervisor
        strategy:     testStrat1
        string hello: "world"

process test2Supervisor:
        pid:      /root/user/test2-supervisor
        strategy: testStrat2

So there you can see that strategies are declared separately to the processes, which allows re-use. Also they refer to global variables which allows for bulk changing of settings across strategies. But you can also use strategies inline:

process test1Supervisor:
    pid:      /root/user/test1-supervisor
    strategy: 
        all-for-one:
            retries: 5
            backoff: min = 1 seconds, max = 100 seconds, step = 2 second
            match
             | LanguageExt.ProcessSetupException -> restart
            redirect when
             | restart  -> forward-to-self