[2015] Drain StackStorm & Rolling Upgrade

[manasdk]

Problem formulation

At the highest level StackStorm needs to be able to support the following -

1. red-black / blue-green upgrade but basically http://cloudnative.io/blog/2015/02/the-dos-and-donts-of-bluegreen-deployment/
2. Rolling upgrade
3. Safe content upgrade on the box.

The actual StackStorm upgrade or content update is not a concern for StackStorm platform. It will be managed by deployment tools that sit outside of StackStorm (whew!). Well then aren't we done; can we go home and play video games yet? Well, no.

StackStorm keeps performing work therefore any external system to be able to handle (1) or (2) correctly StackStorm need to enter a drain state (https://en.wikipedia.org/wiki/drain). Once having successfully entered this state rolling upgrade or content can be upgraded on the box.

Definition of work in the context of ST2

It is best to agree on the definition of work. Following fall under the broad category of work -

1. Running any execution (action or workflow)
2. Triggering timers
3. Accepting webhooks
4. Accepting TriggerInstances from any sensor
5. Internal triggerinstances like actiontrigger, notifytrigger, kvupdate, sensorupdate, sensorexit etc.

Define Drained StackStorm

1. Accept new work but do not perform new work (this really depends on the case)
2. Existing work must continue to completion
3. When/If system is take out of drained state pick-up all the queued up work.

Use-case: Red-Black upgrade

This is specifically to support the NFLX use-case.

Typical StackStorm deployment is -

• 3 identical StackStorm boxes
• Independent RabbitMQ (no cluster)
• Shared DB
• Shared content
• 1 sensor per node with all sensor reading off an SQS queue.

Process

1. start a new upgraded st2 instance and make it part of the cluster i.e. connect sensor to the same Q
2. drain the red node and wait for it to complete work. In this case it is safe for the sensor to simply quit and stop accepting any work.
3. A way to identify that the red node is no longer performing any work and can be safely shutdown
4. continue updating other nodes
5. As 1 node drains or stop accepting any more work other nodes will continue to perform work and therefore must there will be no downtime. It is important to keep this state.

Use-case: Rolling upgrade

Again a multi-node setup where the upgrade of bits happen on every node.

A deployment would be as follows -

• More than 1 StackStorm nodes. Lets assume there are 3.
• Each node run some or all StackStorm services.
• DB is shared
• RabbitMQ is shared

Process

1. Drain node 1 (stop accepting work)
2. Other node continue to perform work.
3. All ActionRunners can be immediately shutdown following the completion of the current execution.
4. Notify or make drained state queriable
5. Services are stopped
6. Bits are upgraded
7. Services are restarted and work starts again

Use-case: Content upgrade

Process

1. drain StackStorm and wait for all work to be drained.
2. It is ok for sensor to accept trigger-instances or for new executions to be accepted so long as they are queued for later execution. (we may want to relax this requirement)
3. Have a way to identify that work is entirely drained.
4. replace and register content
5. un-drain the system

Note : Likely that some old execution or trigger-instances that are queued up are no longer valid. This make continuing to accept new work somewhat hairy.

Design and Implementation

The actual Quiescing has separate meaning depending on context -

1. red-black upgrade - no new work is accepted
2. content upgrade - work is accepted and queued for later execution

Enter drain state

Regardless of these nuances we can definitely see that there need to be a way to Signal all process that they need to -

1. stop performing new work
2. potentially queue up new work or completely reject

SIGUSR2 could be a good way to notify all process. However, the down side is we cannot pass in rich information to the processes that they need to stop performing work. Also, the assumption that each stackstorm process is on the same box is an ok assumption for NFLX but not everywhere else - not even in our our deployments.

Control channel built on rabbitmq which can be used to send rich messages to all processes is an option. This would be at the highest level be triggered by an API call e.g. POST @ /admin/state.
This is likely somewhat non-standard and might lead to confusion but since StackStorm is a system of co-operative process with no machine affinity this might be our only option. The benefit of a rich control channel is for many future extensions.

Identify completion

It is important to know when work is complete. Knowing when no new execution is the only point a user can proceed to the next in upgrade or content update.

All executions are easy to track and identify completion. However, knowing when even the whole rule chain (like st2build002) will not fire any longer is done is also important to track. This way we would not end up with pipelines in a half-baked state. (Likely we make some simplifying assumptions)

Exit drain

In case of content upgrade another signal and in case of red-black upgrade new services will be started and old one removed.

[dzimine]

Internal triggerinstances like actiontrigger, notifytrigger, kvupdate, sensorupdate, sensorexit etc.

when it comes to 'uber-flow' - loose definition of logical unit of work consisting on multiple works, connected via triggers - question is where do we stop at flow, or let the whole uber flow to complete (e.g. continue to emit some internal triggers). In example of build system, we may want the whole build to complete. On the other hand, if we manage to queue the internal triggers, than these uber flows may be resumable.

[manasdk]

when it comes to 'uber-flow' - loose definition of logical unit of work consisting on multiple works, connected via triggers - question is where do we stop at flow, or let the whole uber flow to complete

It is going to be a challenge to address this one but we can make some simplifying assumptions.

• It is possible to allow completion of the uber-flow tied with ActionTrigger or NotifyTrigger. Practically the flows are finite and we will always have the system eventually drain out. This solved the build system type setup.
• We could make a simplifying assumption and only allow the current Executions to complete without worrying about the uber-flow. We have somewhat simplistically assumed even in conversation that buffering TriggerInstances should be possible however as I have been mulling over the problem my sense is that buffering is easier said than done. It is likely simpler to fail these requests (internal trigger-instances and external API calls)

I am not picking an option here I am just saying each is possible and come with pros and cons.

[manasdk]

After thinking about this problem for a few days I suspect we would like want to recommend a deployment and upgrade scenario which may or may not look like NFLX red-black deployment and solve the problem in the context of this deployment.

AFAICT we have 3 separate use-cases that need to be solved here and each have subtle variations in requirements. I suspect we will end up solving the problem optimally for 1 of the 3 and end up with an acceptable sub-par solution for the other 2. To end up with acceptable solutions on all parts I feel we must answer 2 questions -

1. How is StackStorm deployed in HA?
2. How is StackStorm content managed and updated?

[Kami]

I already commented on the whole draining / rolling upgrade stuff a lot on Slack, but let me recap it here.

After thinking about this problem for a few days I suspect we would like want to recommend a deployment and upgrade scenario which may or may not look like NFLX red-black deployment and solve the problem in the context of this deployment.

Yes, we will need docs which explain how to perform a rolling upgrade. And we should be opinionated here and support one right way of doing it. There will obviously be some differences if the user uses non "standard" technology - e.g. haproxy instead of nginx or similar, but that's not our problem - we will explain a high level process and we will potentially have some tooling for standard setup, but that's it.

[manasdk]

Discussion video https://www.youtube.com/watch?v=4fLxsDIHsu4

Will post compiled updates with assumptions and decision later.

[dzimine]

Once you finish designing, please play it back to phool, netflix, and spark. Takes time but it's complex and important feature so "test early" aka validate assumption will serve us well. Email/slack/offline is fine.

[manasdk]

Once you finish designing, please play it back to phool, netflix, and spark.

Yes. That is the plan; DoriftoShoes already knows the plan since he helped come up with the requirements. Still will run it by him before ofcourse.

[manasdk]

Plan

The purpose of draining is to support rolling-upgrades and content refreshes. To describe
the approach first the setup.

Non-HA deployment (single box)

• No rolling upgrade supported
• Content-upgrade supported

HA deployment (multi box)

• Rolling upgrade supported
• Content-upgrade supported

details

1. multiple st2api behind a load balancer
2. All processes are redundant or multiple of them can be spawned and co-operate
3. mistral talks to the st2api via LB

StackStorm Upgrade requirements

1. All service can be drained. The specific definition of drain is service specific.
2. Services do not accept any new work.
3. Once drain-ing is complete service shuts down.
4. Service state is introspectable.

Content Upgrade requirements

1. All service can be drained to put them in maintenance mode.
2. StackStorm as a whole will accept new work but will queue up the work.
3. Once drain-ing is complete service maintains stasis.
4. Content is upgraded(registered etc) and services can be taken out of maintenance.
5. Queued up work is not picked up and performed.

Drain + Shutdown

First drain all existing work and then quit.

st2api

• process stops accepting work.
• Disconnects clients connected to stream (open questions for clients)
• Once all streams quit will terminate.

st2rulesengine

• Timer is immediately shutdown
• Only processes internal triggers (specifically actiontrigger and notificationtrigger) as
  the tie in uber-flows
• Likely receives a signal from the last actionrunner to terminate. (needs some thought)

st2sensorcontainer

• Terminate sensors and quit immediately

st2actionrunner

• Finishes all existing executions
• ActionChainRunner quits only after ActionChain is entirely done
• Since all future executions can be serviced by identical actionrunner that are not draining
  it does not have to accept any new work therefore can disconnect it workQ listener.
• On restart should be able to handle execution from previous StackStorm version. Basically,
  the ActionRunner has to be backwards compatible (!)

st2notifier

• Can quit immediately after last triggerinstance is published

st2resultstracker

• Can stop tracking further request and can hand over to its pair.
• This is tricky as resulttrackers are not designed to share so we may need to come up with
  some co-ordination mechanism.

mistral

• External watchdog can get mistral to quit as soon as managing StackStorm identifies that all
  executions are complete.
• Mistral in the near-term will NOT support HA. We will have some runner affinity baked into
  StackStorm.

Possible options

• pause on drain and resume on un-drain
• keep the api capable of accepting executions coming from Mistral
• allow workflows to fail and simply resume <- (for a single box deploy this will be the route and
  on multi-box deployments mistral would be taking to LB and still able to complete)

Drain + Maintenance

Drain existing work, queue up new work but do not actually perform and on exiting maintenance
complete work.

st2api

• Process continues accept execution requests but queues them up for later execution.

st2rulesengine

• timer likely needs to be shutdown - open question.

possible options

• continue to produce trigger instances and have rules trigger
• buffer trigger instances

st2sensorcontainer

• No change

st2actionrunner

• Finishes all existing executions
• Since all future executions can be serviced by identical actionrunner that are not draining
  it does not have to accept any new work therefore can disconnect it workQ listener.
• on exiting maintenance connect back to work Q

st2notifier

• No change

st2resultstracker

• No change

mistral

• Finish work and accept no more work. This should be for free depending on ActionRunner.

Assorted questions

Why do we support maintenance for content upgrade?

It is possible to simply reject work even to support content upgrade. However, in a event drive
world we cannot ask external systems to retry after some time. We have to accept TriggerInstances
and we might as well then accept Executions from the API to simplify for users. This will lead to
delayed execution but no loss of work.

On content upgrade if an action definition changes it may render an execution incorrect.

True. We would have to mark such an execution a failure. No reporting back to the client.

Why is introspection of service state necessary?

For the operator to know is it safe to upgrade the bits or is it safe to upgrade content this
information is essential. We will likely have to come up with an admin API.

Backwards compatibility.

Rolling upgrades makes backwards compatibility a must. e.g. a new ActionRunner must be able to
handle executions started with a previous version of StackStorm. This is as lakshmi-kannan puts
it the hardest requirement.

What happens to red-black deployments?

We will support red-black deployments. In those case we would need to have introspection APIs to
identify if StackStorm is done processing requests.

What if a service refuses to drain?

We can enforce hard timeouts to keep the window of draining reasonable. This will be configurable
and system specific.

Will we have a tool?

Likely we will provide a tool but in the short term try to see how far sending signals to process
and robust docs take us.

[jfryman]

Discussion video https://www.youtube.com/watch?v=4fLxsDIHsu4

Excellent! Anytime we have to go high bandwidth like this, I think it's wonderful to capture so that others can catch up async.

Any chance for a small set of docs on how to set this up?

[manasdk]

Any chance for a small set of docs on how to set this up?

will do

[Kami]

Yeah, the latest proposal in more in line with what I had in mind 👍

Some minor stuff:

Disconnecting stream clients & long running requests

We can add support for "disconnect" message which we send over stream. This message would tell the client to disconnect from the stream endpoint (and potentially reconnect since API should be behind load balancer and other non-draining API server will process the request).

Keep in mind that this is really just an optimization and we still need a hard timeout and disconnect all the clients if the timeout has been reached (e.g. stuck or none cooperative client, etc.). That's also what we do right now.

[Kami]

Here is just a quick rant from Slack about the whole resume thing and managing users expectations (and being opinionated) - https://stackstorm.slack.com/archives/general/p1447931039000274.

I will recap it in a more readable and less ranty form later :P

kami [12:03]
yeah, surviving chaos monkey should be our goal as well. drain is nice for rolling upgrade but in real word things can go away / die anytime. the thing which is the hardest to solve here is long running workflows and actions
​[12:04]
and tbh, we can't and really shouldn't be trying to fully solve it, it's a distributed system, there are limitations
​[12:04]
so we need to push work on the client - this really means users who want some properties should keep some things in mind when designing workflows
​[12:05]
most notably making actions and flows idempotent, that's the most important thing
​[12:06]
that's just reality of a distributed system, trying to do any kind of "smart resume" or similar will just result on tons of stupid code to maintain and there will still be too many edge cases and wont solve the root problem since you cant change the laws of physics
🚀1
​[12:06]
so biggest step is documentation - informing the user and developers
​[12:06]
it's the same with micro services (hate that word) - a lot of stuff is now on the developer, that sucks, and there are things you can do to move some of that stuff away, but some if it will always be on the developer
​[12:07]
so i also want to make sure we wont promise something which is not even possible 😛
​[12:11]
but i would also imagine a lot of people at netflix are already aware of that - so as long as we document that well - we need a section on good patterns and anti patterns for flows (and actions and pretty much everything else) and we add improvements to the platform we should be fine
​[12:13]
so again, i also want to make sure, we, ourselves are looking at the solution from the right perspective - and not trying to solve a thing which is not unsolvable (e.g. if action runner gets killed while it's execution action composed of multiple steps - there is literally nothing we can do to solve that, even if we introduced some crazy check points or whatever, this wouldnt fully solve the problem and would make writing actions way more complicated. imo the right idea is, make actions idempotent, if it fails part way through, we will simply restart / re-run that action - that's something we can do)
epowell101 [13:29]
A few things I like here a) simplicity and hence grokabikity over incredible / impossible functionality b) establishing patterns is massively cheaper than writing and maintaining code c) gets main point which is even / esp developers find learning curve a bit much and we need to decrease that burden as a priority.
​[13:31]
In storage we had to know the CAP theorem well enough to cut through magic BS from competitors and our users; similar argument here w/ trade offs here I'd imagine.
​[13:32]
Feels like excellent content for "someone" to write w help for docs, maybe FAQ, and blog as well.
kami [13:42]
agreed, it's the same with some anti patterns we seen with actions - there are many ways to make something work, but not all the ways are the same (some are harder to maintain, test, etc)
kami [14:03]
it also ties into the whole "legacy" vs new style (docker / micro services) paas
​[14:03]
with legacy paas and technologies like nas, there are limitations and you cant do everything
​[14:03]
which is fine if you are aware of that at the begining
​[14:04]
and there is still tons of place for legacy paases, after all, there are still tons of legacy apps where rebuilding it doesnt make sense or it's too expensive
​[14:04]
and trying to do both, imo is a bad idea
​[14:04]
both in a single product
​[14:04]
it causes confusion
​[14:04]
and people will expect they get some stuff for free
​[14:04]
but they wont since the followed "legacy" guidelines
​[14:05]
so having said that, what i really means is that we need to position ourselves and manage expectations
​[14:05]
you want HA? well, you need to design workflows and actions in that manner
​[14:05]
but still, like i said above, giving people choice in many cases is bad
​[14:05]
a lot of legacy paases wanted to do that
​[14:05]
didnt work out for them
​[14:05]
since they werent opinionated enough
​[14:05]
and people result to same old legacy technologies they knew so they didnt get much out of it
​[14:06]
anyway, will try to spend some time this weekend on action and workflow patterns and anti patterns blog post