query scheduler: querier-worker queue priority algo implementation (#…

…9013)

Example results of benchmark test `TestMultiDimensionalQueueAlgorithmSlowConsumerEffects`:

```
Results by query component:
tree: tenant-querier -> query component round-robin tree, 1 tenant, 10pct slow queries: seconds in queue: [ingester: mean: 0.0499 stddev: 0.03 store-gateway: mean: 0.0029 stddev: 0.00]
tree: query component round-robin -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue: [ingester: mean: 0.0960 stddev: 0.03 store-gateway: mean: 0.0017 stddev: 0.00]
tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue: [ingester: mean: 0.0162 stddev: 0.01 store-gateway: mean: 0.0098 stddev: 0.01]
tree: tenant-querier -> query component round-robin tree, 1 tenant, 25pct slow queries: seconds in queue: [ingester: mean: 0.1912 stddev: 0.06 store-gateway: mean: 0.0806 stddev: 0.07]
tree: query component round-robin -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue: [ingester: mean: 0.1942 stddev: 0.06 store-gateway: mean: 0.0894 stddev: 0.08]
tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue: [ingester: mean: 0.0132 stddev: 0.01 store-gateway: mean: 0.1086 stddev: 0.08]
tree: tenant-querier -> query component round-robin tree, 1 tenant, 50pct slow queries: seconds in queue: [ingester: mean: 0.3635 stddev: 0.18 store-gateway: mean: 0.2155 stddev: 0.15]
tree: query component round-robin -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue: [ingester: mean: 0.3618 stddev: 0.18 store-gateway: mean: 0.2165 stddev: 0.15]
tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue: [ingester: mean: 0.0096 stddev: 0.00 store-gateway: mean: 0.2264 stddev: 0.15]
tree: tenant-querier -> query component round-robin tree, 1 tenant, 75pct slow queries: seconds in queue: [ingester: mean: 0.2160 stddev: 0.15 store-gateway: mean: 0.3468 stddev: 0.23]
tree: query component round-robin -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue: [ingester: mean: 0.1921 stddev: 0.14 store-gateway: mean: 0.3580 stddev: 0.23]
tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue: [ingester: mean: 0.0049 stddev: 0.00 store-gateway: mean: 0.3402 stddev: 0.22]
tree: tenant-querier -> query component round-robin tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.0528 stddev: 0.05 store-gateway: mean: 0.4271 stddev: 0.27]
tree: query component round-robin -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.0601 stddev: 0.06 store-gateway: mean: 0.4246 stddev: 0.27]
tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.0022 stddev: 0.00 store-gateway: mean: 0.4241 stddev: 0.27]
tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue: [ingester: mean: 0.2189 stddev: 0.15 store-gateway: mean: 0.2193 stddev: 0.15]
tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue: [ingester: mean: 0.3614 stddev: 0.18 store-gateway: mean: 0.2177 stddev: 0.15]
tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue: [ingester: mean: 0.0094 stddev: 0.00 store-gateway: mean: 0.2237 stddev: 0.15]
tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.2933 stddev: 0.14 store-gateway: mean: 0.2035 stddev: 0.14]
tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.3626 stddev: 0.18 store-gateway: mean: 0.2141 stddev: 0.15]
tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.0088 stddev: 0.00 store-gateway: mean: 0.2286 stddev: 0.16]
tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue: [ingester: mean: 0.3601 stddev: 0.18 store-gateway: mean: 0.2202 stddev: 0.15]
tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue: [ingester: mean: 0.3633 stddev: 0.18 store-gateway: mean: 0.2152 stddev: 0.15]
tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue: [ingester: mean: 0.0093 stddev: 0.00 store-gateway: mean: 0.2276 stddev: 0.16]

Results for ingester-only queries by tenant ID:
tree: tenant-querier -> query component round-robin tree, 1 tenant, 10pct slow queries: seconds in queue:[tenant-0: mean: 0.0499 stddev: 0.03]
tree: query component round-robin -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue:[tenant-0: mean: 0.0960 stddev: 0.03]
tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue:[tenant-0: mean: 0.0162 stddev: 0.01]
tree: tenant-querier -> query component round-robin tree, 1 tenant, 25pct slow queries: seconds in queue:[tenant-0: mean: 0.1912 stddev: 0.06]
tree: query component round-robin -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue:[tenant-0: mean: 0.1942 stddev: 0.06]
tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue:[tenant-0: mean: 0.0132 stddev: 0.01]
tree: tenant-querier -> query component round-robin tree, 1 tenant, 50pct slow queries: seconds in queue:[tenant-0: mean: 0.3635 stddev: 0.18]
tree: query component round-robin -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue:[tenant-0: mean: 0.3618 stddev: 0.18]
tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue:[tenant-0: mean: 0.0096 stddev: 0.00]
tree: tenant-querier -> query component round-robin tree, 1 tenant, 75pct slow queries: seconds in queue:[tenant-0: mean: 0.2160 stddev: 0.15]
tree: query component round-robin -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue:[tenant-0: mean: 0.1921 stddev: 0.14]
tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue:[tenant-0: mean: 0.0049 stddev: 0.00]
tree: tenant-querier -> query component round-robin tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0528 stddev: 0.05]
tree: query component round-robin -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0601 stddev: 0.06]
tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0022 stddev: 0.00]
tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue:[tenant-0: mean: 0.2355 stddev: 0.15 tenant-1: mean: 0.0483 stddev: 0.05]
tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue:[tenant-0: mean: 0.3894 stddev: 0.17 tenant-1: mean: 0.0924 stddev: 0.06]
tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0101 stddev: 0.00 tenant-1: mean: 0.0031 stddev: 0.00]
tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.3322 stddev: 0.11 tenant-1: mean: 0.1902 stddev: 0.13]
tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.4159 stddev: 0.15 tenant-1: mean: 0.1980 stddev: 0.17]
tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.0100 stddev: 0.00 tenant-1: mean: 0.0049 stddev: 0.00]
tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue:[tenant-0: mean: 0.3376 stddev: 0.19 tenant-1: mean: 0.3814 stddev: 0.17]
tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue:[tenant-0: mean: 0.3538 stddev: 0.19 tenant-1: mean: 0.3730 stddev: 0.17]
tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue:[tenant-0: mean: 0.0090 stddev: 0.00 tenant-1: mean: 0.0097 stddev: 0.00]
```

pkg/frontend/v1/frontend.go

@@ -239,14 +239,15 @@ func (f *Frontend) Process(server frontendv1pb.Frontend_ProcessServer) error {
 	}
 	defer f.requestQueue.SubmitUnregisterQuerierWorkerConn(querierWorkerConn)
 
-	lastTenantIndex := queue.FirstTenant()
+	lastTenantIdx := queue.FirstTenant()
 
 	for {
-		reqWrapper, idx, err := f.requestQueue.WaitForRequestForQuerier(server.Context(), lastTenantIndex, querierID)
+		dequeueReq := queue.NewQuerierWorkerDequeueRequest(querierWorkerConn, lastTenantIdx)
+		reqWrapper, idx, err := f.requestQueue.AwaitRequestForQuerier(dequeueReq)
 		if err != nil {
 			return err
 		}
-		lastTenantIndex = idx
+		lastTenantIdx = idx
 
 		req := reqWrapper.(*request)
 
@@ -266,7 +267,7 @@ func (f *Frontend) Process(server frontendv1pb.Frontend_ProcessServer) error {
 		  it's possible that its own queue would perpetually contain only expired requests.
 		*/
 		if req.originalCtx.Err() != nil {
-			lastTenantIndex = lastTenantIndex.ReuseLastTenant()
+			lastTenantIdx = lastTenantIdx.ReuseLastTenant()
 			continue
 		}