consistent feature toggles across a single request life cycle

[adamluzsi]

Context

When various components call the centralized release service multiple times across a single request, it becomes possible to receive a different flag state back during a single life cycle of a request.

Such use-case is when the customer uses a microservice architecture where release toggles were not part of the design initially. passing around feature toggle states becomes change with high cost. They would like to use the service without the need to update all the microservice to pass around this new data.

Proposal

Since the 12factor app conventions, many if not all distributed systems use some form of request tracing ID.

E.g.:

• Heroku: X-Request-ID
• AWS: X-Amzn-Trace-Id: Root=...

Using this tracing ID we can make a special kind of cache that is optimized for high throughput write/read operations for /v/config endpoint.

The scaling would be done on a per tracing-id mod cluster

[adamluzsi]

@DavudSafarli

[adamluzsi]

I thought about this, and I think using a tracking ID would be a mistake.
If we use a tracking ID we will be forced to have a cache that holds the tracking ID's state till a certain time defined as SLA.
This would both decrease the scalability and increase the system complexity,

We should have a timestamp value instead, that defines the start of the tracked request event.
With that, we can easily use the existing system setup, and no need to introduce a new non-distributed caching layer.
The only thing we should consider is to convert the rollouts table into a log table, where by default we would always look at the latest value. This can be really easily achieved by using the block range index database indexing technique on the created_at/event_time column.

What do you think @DavudSafarli?

[adamluzsi]

Based on our phone conversations here is the latest iteration of the design.

Toggler Tracking Token

This tracking token allows the calling system to create a read-only transactional state about the token
that allows toggler-users to have consistent release states across all callers, to any release rollout value.

high-level workflow:

• toggler-tracking-token requested with contextual information
• toggler-tracking-token used in all parts of the request to make release flags express a consistent state during a form workflow.
• all the callers with toggler-tracking-token will also provide contextual information about the end-user for the rollout strategy.

use-cases

No modification during the flow

@startuml

title Transactional Toggler Tracking Token

actor "user/client" AS usr
participant "UI" AS ui
participant "BE" AS be
participant "Toggler" AS tg

usr -> ui : starts a workflow
alt if meta data is sensitive
	ui ->  be : request tracking-token
	be ->  tg : request tracking-token, including all meta data that might be used during
	tg --> be : return (signed) tracking-token
	be --> ui : return tracking-token
else if meta data in the tracking token isn't sensitive
	ui ->  tg : request tracking-token, including all meta data that might be used during
	tg --> ui : return (signed) tracking token
end
ui --> usr :
usr -> ui  : process interacting with a certain workflow form
ui  -> be  : send request including toggler tracking-token
be  -> tg  : request release states, etc with tracking-token
tg  -> tg  : unwrap token, verify which release rollout state required to evaluate the release flag states
tg --> tg  : resolve release flag, config, etc states
note left
	the tracking token contains optional information that the tracking-token requester provides
	The provided contextual information is kept in the tracking token.
	Each time a release flag is requested, the rollout strategy will receive the same metadata from the tracking token.
	No matter what happens with the rollout state, the same workflow remains consistent and will use the same states.
	If a release rollout is deleted/modified during the usage of the tracking token,
	the release state is still guaranteed to yield the same results.
	This is also true for release states that were not requested beforehand.
		........................................................................................................................................................................
end note
tg --> be  : return states
be ->  be  : process depending on the states
be --> ui  : return

@enduml

release modified (deleted/updated) during the usage of toggler-tracking-token

@startuml

title Transactional Toggler Tracking Token

actor "user/client" AS usr
participant "UI" AS ui
participant "BE" AS be
participant "Toggler" AS tg
participant "Toggler GUI" AS tggui


usr -> ui : starts a workflow
alt if meta data is sensitive
	ui ->  be : request tracking-token
	be ->  tg : request tracking-token, including all meta data that might be used during
	tg --> be : return (signed) tracking-token
	be --> ui : return tracking-token
else if meta data in the tracking token isn't sensitive
	ui ->  tg : request tracking-token, including all meta data that might be used during
	tg --> ui : return (signed) tracking token
end
ui --> usr :
usr -> ui  : process interacting with a certain workflow form
ui  -> be  : send request including toggler tracking-token
par
	tggui -> tg    : modify (delete/update) rollout strategy.
	tg   --> tggui : done
end
be  -> tg  : request release states, etc while using the tracking-token
tg  -> tg  : unwrap token, verify which release rollout state required to evaluate the release flag states
tg --> tg  : resolve release flag, config, etc states, use rollout state that matches the tracking-token's creation time
tg --> be  : return states according to the tracking-token
be ->  be  : process depending on the states
be --> ui  : return

@enduml

Let me know @DavudSafarli when you have further observations on the design.

Cheers,
Adam

[DavudSafarli]

So, just to rephrase to see if I understood correctly, the idea is, given that token (that self-contains the timestamp of the client's first interaction with Toggler during the request lifecycle) is provided, always evaluate the Flag State according to the Rollout Plan that existed at the time that token issued.

And for this, you are suggesting that, Storage should be able to remember the state(config) of the same Rollout at different times, instead of just the current state of it.

If my understanding is correct till this point, then I think the current solution looks simple(at least in theory:D) and good. I'll think throughout the day whether anything else is possible or if anything is missing(as I said, I feel like something does :D)

[adamluzsi]

@DavudSafarli by current solution, do you mean the current implementation on master or the latest iteration of the design proposal?