Perf test dashboard setup (#212)

* Adding perf test component
* Adding bicep templates for creating perf test dashboard
* Adding authentication for prometheus pushgateway
* Adding dapr perf test azure resources bicep
* perf-test resoruce creation
* Update deploy/aks/monitoring/perf/README.md

--------

Signed-off-by: MD Ashique <[email protected]>
Co-authored-by: Tiago Alves Macambira <[email protected]>

deploy/aks/monitoring/perf/README.md

@@ -0,0 +1,116 @@
+## Performance Test Dashboard Setup
+
+This document serves as a guideline for setting up a performance test dashboard using Azure managed Prometheus and Grafana. We will be creating an AKS (Azure Kubernetes Service) cluster and installing Prometheus Pushgateway and Azure Monitoring Agent (AMA) to scrape performance test metrics and push them to Azure managed Prometheus. Additionally, we will set up an ingress controller with authentication to act as a proxy for Prometheus Pushgateway. Finally, we will create a Grafana dashboard by importing a predefined JSON model.
+
+Below are step-by-step instructions to set up a performance test dashboard using Azure managed Prometheus and Grafana. Follow these steps to configure your performance monitoring environment.
+
+#### Step 1: Clone the test-infra repo
+
+```bash
+git clone https://github.com/dapr/test-infra.git
+cd deploy/aks/monitoring/perf
+```
+
+#### Step 2: Login to Azure and set your subscription
+
+```bash
+az login
+export SUBSCRIPTION_ID=<SUBSCRIPTION UUID TO BE USED FOR THE COMMANDS BELLOW>
+az account set --subscription "${SUBSCRIPTION_ID}"
+```
+
+### Step 3: Set resource group name, location, cluster name and use the same namespace (dapr-perf-metrics)
+
+```bash
+DAPR_PERF_RG=<resource group to be used>  
+DAPR_PERF_LOCATION=<insert region>  
+CLUSTER_NAME=<cluster name>
+DAPR_PERF_METRICS_NAMESPACE=dapr-perf-metrics
+```
+
+#### Step 4: Create Resource Group
+```bash
+az group create --name $DAPR_PERF_RG --location $DAPR_PERF_LOCATION
+```
+
+#### Step 5: Execute main.bicep and provide AKS cluster name on prompt
+
+```bash
+az deployment group create --resource-group $DAPR_PERF_RG --template-file main.bicep --parameters clusterName="${CLUSTER_NAME}"
+```
+
+#### Step 6: Merge Newly Created Cluster Username and Password
+
+```bash
+az aks get-credentials --resource-group $DAPR_PERF_RG --name "${CLUSTER_NAME}"
+```
+
+#### Step 7: Switch AKS Cluster Context
+
+```bash
+kubectl config use-context "${CLUSTER_NAME}"
+```
+
+#### Step 8: Install Prometheus Pushgateway
+
+```bash
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm repo update
+helm upgrade --install \
+    prometheus-pushgateway prometheus-community/prometheus-pushgateway \
+    --namespace $DAPR_PERF_METRICS_NAMESPACE \
+    --wait
+```
+
+#### Step 9: Install Ingress Controller. 
+
+Follow this [link](https://learn.microsoft.com/en-us/azure/aks/ingress-basic?tabs=azure-cli#basic-configuration) for more details on setting up nginx ingress controller.
+
+```bash
+helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
+helm repo update
+
+helm upgrade --install \
+    ingress-nginx ingress-nginx/ingress-nginx \
+  --namespace $DAPR_PERF_METRICS_NAMESPACE \
+  --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz
+```
+
+#### Step 10: Create Username and Password for Authentication
+
+To create a basic authentication [username and password](https://kubernetes.github.io/ingress-nginx/examples/auth/basic/), use the following command, which will create an auth file and prompt you to provide a username and password.
+
+```bash
+htpasswd -c auth <user name for prometheus pushgateway>
+```
+
+#### Step 11: Create a Secret in Kubernetes
+
+```bash
+kubectl create secret generic basic-auth --from-file=auth -n dapr-perf-metrics
+```
+
+#### Step 12: Create Ingress for Prometheus Pushgateway
+
+```bash
+kubectl apply -f ./prometheus-pushgateway-ingress.yaml
+```
+
+#### Step 13: Create a Config Map for Service Discovery for AMA Agent
+
+```bash
+kubectl apply -f ./prometheus-pushgateway-configmap.yaml
+```
+
+#### Step 14: Add user to grafana
+
+- Go to grafana resource in Azure portal
+- Select Access control (IAM) on left menu
+- Click on Add role assignment
+- Select suitable role for the user, and click Next
+- In the Member tab, click on `+ Select Member` and type their email in search box
+- Select user and click on `Review + assign`
+
+#### Step 15: Create Grafana Dashboard
+
+Grab the granfa link from azure portal and create a Grafana dashboard by importing the [JSON model](https://github.com/dapr/dapr/blob/78b7271f015fa935fd59299357787f3e86861300/tests/grafana/grafana-perf-test-dashboard.json). Ensure to update all [`uid` of `datasource`](https://github.com/dapr/dapr/blob/78b7271f015fa935fd59299357787f3e86861300/tests/grafana/grafana-perf-test-dashboard.json#L41) objects present in the JSON file to match your configuration.

deploy/aks/monitoring/perf/main.bicep

@@ -0,0 +1,104 @@
+
+//
+// Global parameters
+//
+
+@minLength(5)
+@maxLength(30)
+@description('Name of the cluster')
+param clusterName string
+
+@minLength(1)
+@maxLength(3)
+@description('Short, up to 3 letters prefix to help identify cluster resources. Lowercase and numbers only')
+param shortClusterPrefixId string = 'pt'
+
+
+@description('The unique discriminator of the solution. This is used to ensure that resource names are unique.')
+@minLength(3)
+@maxLength(16)
+param solutionName string = toLower('${shortClusterPrefixId}${uniqueString(resourceGroup().id)}')
+
+// Per cluster resources
+param identityName string = '${solutionName}-identity'
+@minLength(2)
+@maxLength(24)
+param grafanaName string = '${solutionName}-grafana'
+param amwName string = '${solutionName}-amw'
+param logAnalyticsName string = '${solutionName}-la'
+
+// Safe defaults
+param agentVMSize string = 'standard_d2s_v5'
+param location string = resourceGroup().location
+param kubernetesNamespace string = 'dapr-perf-metrics'
+@description('ObjectID for an user in AAD you want to grant grafana admin rights. Default is to not provide anything: not grant this permission any individual')
+param userGrafanaAdminObjectId string = ''
+
+
+
+// Identity - Not a module so we can reference the resource below.
+resource managedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
+  name: identityName
+  location: location
+}
+
+//
+// Cluster infrastructure
+//
+
+
+// AKS Cluster - Not a module so we can reference the resource below.
+resource aks 'Microsoft.ContainerService/managedClusters@2023-03-01' = {
+  name: clusterName
+  location: location
+  identity: {
+    type: 'UserAssigned'
+    userAssignedIdentities: {
+      '${managedIdentity.id}': {}
+    }
+  }
+  properties: {
+    dnsPrefix: '${clusterName}-dns'
+    agentPoolProfiles: [
+      {
+        name: 'agentpool'
+        count: 3
+        vmSize: agentVMSize
+        osType: 'Linux'
+        mode: 'System'
+      }
+    ]
+    azureMonitorProfile: {
+      metrics: {
+        enabled: true
+        kubeStateMetrics: {
+          metricLabelsAllowlist: '*'
+        }
+      }
+    }
+  }
+}
+
+// Apply the k8s namespace - applications live here
+module daprPerfNamespace '../../services/namespace.bicep' = {
+  name: '${clusterName}--namespace'
+  params: {
+    kubeConfig: aks.listClusterAdminCredential().kubeconfigs[0].value
+    kubernetesNamespace: kubernetesNamespace
+  }
+}
+
+module monitoring '../monitoring.bicep' = {
+  name: '${clusterName}--monitoring'
+  params: {
+    location: location
+    clusterName: clusterName
+    dceName: '${clusterName}-dce'
+    dcrName: '${clusterName}-dcr'
+    grafanaName: grafanaName
+    workspaceAzureMonitorName: amwName
+    workspaceLogAnalyticsName: logAnalyticsName
+    grafanaAdminObjectId: managedIdentity.properties.principalId
+    userGrafanaAdminObjectId: userGrafanaAdminObjectId
+  }
+}

deploy/aks/monitoring/perf/prometheus-pushgateway-configmap.yaml

@@ -0,0 +1,15 @@
+kind: ConfigMap
+apiVersion: v1
+data:
+  prometheus-config: |-
+    global:
+      scrape_interval: 300s
+    scrape_configs:
+    - job_name: pushgateway
+      scheme: http
+      metrics_path: /metrics
+      static_configs:
+      - targets: ['prometheus-pushgateway.dapr-perf-metrics.svc.cluster.local:9091']
+metadata:
+  name: ama-metrics-prometheus-config
+  namespace: kube-system

deploy/aks/monitoring/perf/prometheus-pushgateway-ingress.yaml

@@ -0,0 +1,24 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: prometheus-pushgateway-ingress
+  namespace: dapr-perf-metrics
+  annotations:
+    # type of authentication
+    nginx.ingress.kubernetes.io/auth-type: basic
+    # name of the secret that contains the user/password definitions
+    nginx.ingress.kubernetes.io/auth-secret: basic-auth
+    # message to display with an appropriate context why the authentication is required
+    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required to push metrics to prometheus-pushgateway'
+spec:
+  ingressClassName: nginx
+  rules:
+  - http:
+      paths:
+      - pathType: Prefix
+        path: "/"
+        backend:
+          service:
+            name: prometheus-pushgateway
+            port:
+              number: 9091