Skip to content

Commit

Permalink
backend: Rewrite Cosmos DB scanning
Browse files Browse the repository at this point in the history
In anticipation of the Cosmos DB containers being merged and
partitioned by subscription ID, this commit fully rewrites how
the backend finds and processes operation documents in Cosmos DB.

Instead of querying for all operation documents across all Azure
subscriptions and processing them serially, the backend now uses
a goroutine-based "worker pool" where each worker is responsible
for processing operation documents within a single subscription/
(soon-to-be) Cosmos DB partition.

This dramatically increases the parallelism within the backend
without significantly changing the Cluster Service interaction
logic.

The polling intervals and worker pool size within the backend
may require further tuning.
  • Loading branch information
Matthew Barnes committed Feb 3, 2025
1 parent a8ad3e2 commit 5f4bb84
Show file tree
Hide file tree
Showing 2 changed files with 220 additions and 118 deletions.
Loading

0 comments on commit 5f4bb84

Please sign in to comment.