Denormalized Policies

Proposal

Current State

Currently policies have a set of roles. These are evaluated and the corresponding links are created between the policies and corresponding entities. This evaluation is done whenever a policy changes, when a related entity is created or updated.

Example: service-policy has identityRoles and serviceRoles. These get evaluated into bidirectional links from service policy to identity and services. When identityRoles or serviceRoles change, they are re-evaluated. When identities or services are created they are evaluated against all service policies. If identity or service roleAttributes change, they are also re-evaluated.

Future State

Fully denormalize policy relationships. For example, instead of identities being only linked to service policies and edge router policies, they will also be linked to the services and edge routers that they have access to via those policies. We also want to introduce policy events so we can react when an identity gains or loses access to services or edge routers.

Finally because we could be creating very large numbers of records, we want to make processing asynchronous so that we don't lock the datastore for extended periods.

Implementation

1. Update link collection API so it indicates when we have added or removed entries from link collection. This will let us know when a policy evaluation results in a change.
2. When a policy evaluation results in a change, update denormalized links.
3. Because entities can have links via multiple policies will need to implement reference counting to ensure we only remove a link when no more policies reference it.
4. Need to ensure deletes work correctly.

For ref counting, we already track maps.

identities//services/ -> nil identities//services/ -> nil

We can create an API which puts the ref count in place of the nil.

Async

Create a channel based API which takes operations. Back the channel with datastore entries in case of abrupt system failure.

• eventSeq=1,eventType=identityCreated,identityId=,policyCursor=
• eventSeq=2,eventType=servicePolicyIdentityRolesUpdated,policyId=,identityCursor=
• eventSeq=3,eventType=identityAddedToServicePolicy,policyId=,identityId=,serviceCursor=

Will need events for entity created/updated, policies created, updated and entity added/removed from policy. If we track progress we can also split processing up into batches.