[New Hack] 054-Azure Load Testing (microsoft#479)

* Created WhatTheHack template stub

* Adding first challenge to review

* Updating challenge 1

* Initial commit of challenge 3/5/7

* Set theme jekyll-theme-slate

* Updating the challenges with first run through.

* Updated challenges

* Updated challenges

* Updated challenges

* Updated challenges

* Updated challenges

* Updated challenges

* Adding resource for challenge 1

* Updating link to github project which still has install instructions

* Solution for challenge 2

* Updated challenges

* Set theme jekyll-theme-midnight

* Updating challenge 2/3 to include creating jmeter script and moving azure load test to challenge 3

* Fix some spacing

* Moving the student resource to challenge 1 markdown file as this is now a simple app that can be described

* Updating the solution to challenge 1

* Pushing solution 2 to coaches guide

* Updating challenge 3 and adding solution

* Updated challlenge

* Updated challenges

* Update README.md

* Checking in solution for challenge 4

* Updating solution 4 markdown file

* Adding solution 5

* Updating links and insights

* Update README.md

* Update Challenge-02.md

* Update Challenge-02.md

* Update Challenge-02.md

* Updating challenge 6 and adding solution

* Adding solution 7

* Update README.md

* Update Challenge-04.md

* Update Challenge-03.md

* Update Solution-01.md

* Renaming and updating script to use paramteres for number of users

* Updating spelling errors

* Update README.md

Removed repository contents section

* Gramatical updates along with updates to solution readme

* Updating with challenge 0 and sample application

* Removing link to sub project

* Adding sample application

* Updating to include hack number

* Updating WTH with feedback from initial review

* Updating verbiage

* updating link to work

* Updating solution 2

* Updating based on WTH review comments

* Updating based on feedback from WTH team

* Removing duplicate text in coach readme

* Adding additional context for the student/resources folder

* Updating verbiage from WTH review

Co-authored-by: GitHub Actions Bot <>
Co-authored-by: Kevin M. Gates <[email protected]>

054-AzureLoadTesting/Coach/README.md

@@ -0,0 +1,64 @@
+# What The Hack - Azure Load Testing - Coach Guide
+
+## Introduction
+Welcome to the coach's guide for the Azure Load Testing What The Hack. Here you will find links to specific guidance for coaches for each of the challenges.  
+
+NOTE: If you are a Hackathon participant, this is the answer guide. Don't cheat yourself by looking at these during the hack! Go learn something. :)
+
+## Coach's Guides
+- Challenge 00: **[Prerequisites - Ready, Set, GO!](./Solution-00.md)**
+	 - Prepare your workstation to work with Azure.
+- Challenge 01: **[Develop a Load Testing Strategy](./Solution-01.md)**
+	 - How to develop a load testing strategy for your application
+- Challenge 02: **[Create a Load Testing Script](./Solution-02.md)**
+	 - Deploy a sample application and create JMeter scripts to support your load testing strategy
+- Challenge 03: **[Create Azure Load Testing Service and Establish Baselines](./Solution-03.md)**
+	 - Create Azure Load Testing Service and learn techniques on how to establish baselines for your application
+- Challenge 04: **[Enable Automated Load Testing (CI/CD)](./Solution-04.md)**
+	 - Incorporate load testing into your CI/CD Pipeline
+- Challenge 05: **[Identify & Remediate Bottlenecks](./Solution-05.md)**
+	 - Review load test results and identify bottlenecks
+- Challenge 06: **[Stress Testing](./Solution-06.md)**
+	 - How to perform stress tests and observing your application behavior
+- Challenge 07: **[Load Testing With Chaos Experiment (Resilience Testing)](./Solution-07.md)**
+	 - Incorporate load testing and chaos experiments together
+
+## Coach Prerequisites
+
+This hack has pre-reqs that a coach is responsible for understanding and/or setting up BEFORE hosting an event. Please review the [What The Hack Hosting Guide](https://aka.ms/wthhost) for information on how to host a hack event.
+
+The guide covers the common preparation steps a coach needs to do before any What The Hack event, including how to properly configure Microsoft Teams.
+
+### Student Resources
+
+Before the hack, it is the Coach's responsibility to download and package up the contents of the `/Student/Resources` folder of this hack into a "Resources.zip" file as this contains the sample application they will need for this hack. The coach should then provide a copy of the Resources.zip file to all students at the start of the hack.
+
+Always refer students to the [What The Hack website](https://aka.ms/wth) for the student guide: [https://aka.ms/wth](https://aka.ms/wth)
+
+**NOTE:** Students should **not** be given a link to the What The Hack repo before or during a hack. The student guide does **NOT** have any links to the Coach's guide or the What The Hack repo on GitHub.
+
+## Azure Requirements
+
+This hack requires students to have access to an Azure subscription/resource group where they can create and consume Azure resources. These Azure requirements should be shared with a stakeholder in the organization that will be providing the Azure subscription(s) that will be used by the students.
+
+Requirements:
+
+- Contributor permissions to a subscription or resource group
+- Resources - The following resources will be created during the hack
+	- Azure Load Testing
+	- Azure App Service
+	- Azure Cosmos DB
+	- Azure Application Insights
+
+## Suggested Hack Agenda
+This hack is estimated to take 3 days with 4 hours each day.
+## Repository Contents
+
+- `./Coach`
+  - Coach's Guide and related files
+- `./Coach/Solutions`
+  - Solution files with completed example answers to a challenge
+- `./Student`
+  - Student's Challenge Guide
+- `./Student/Resources`
+  - Sample app code, ARM Template, and deployment script is located here for students to deploy the sample app used in the challenges  (Must be packaged up by the coach and provided to students at start of event)

054-AzureLoadTesting/Coach/Solution-00.md

@@ -0,0 +1,7 @@
+# Challenge 00 - Prerequisites - Ready, Set, GO! - Coach's Guide 
+
+**[Home](./README.md)** - [Next Solution >](./Solution-01.md)
+
+## Notes & Guidance
+
+We are only setting up the students machines here.  Please reference the student section.  Feel free to assist more during the prerequisite. [Student Pre-reqs.](../Student/Challenge-00.md)

054-AzureLoadTesting/Coach/Solution-01.md

@@ -0,0 +1,39 @@
+# Challenge 01 - Develop a Load Testing Strategy - Coach's Guide 
+
+[< Previous Solution](./Solution-00.md) - **[Home](./README.md)** - [Next Solution >](./Solution-02.md)
+
+## Notes & Guidance
+
+Below is a sample load testing plan we have mapped out.  Students will have something different as we left it up to the student to interpret the environment however the overall feel should be similar.
+
+- Sample Load Testing Plan
+
+
+- Define what services and the scope of your testing - Requirements
+    - We will be simulating an average day when a user hits our website during normal business hours 8 - 5 PM.  We will be testing all 3 endpoints (Get/Add/lasttimestamp) to ensure all features are tested.  
+
+- Environment Requirements
+    - Since we are testing the user, we will need the following resources at the same level as production.
+        - App Service
+        - Cosmos DB
+
+
+- Identify the load characteristics and scenario
+    - Our current average daily load is 1000 users/hour.  We will mimic the average daily load hitting all 3 endpoints.  Since there is consistent daily load we will mimic this with gradual load increasing linearly.  
+
+- Identify the test failure criteria
+    - Our test will fail if we are unable to support 1000 users as we already know this is the current demand.
+    - We will also consider this a failed test if we find performance below expected thresholds as this may impact customer satisfaction.
+
+- Identify how you will be monitoring your application
+    - We will be monitoring our applications with Application Insights to detect any errors and monitor performance.
+
+- Identify potential bottlenecks/limitations in advance
+    - Since we do not cache any data at any level.  Any component failing could cause an issue downstream.
+
+- Please note any assumptions
+    - We are assuming these are only casual end users and not other 3rd parties who could be trying to obtain information about our traffic.
+    - We are using consumption based model for Azure App Services
+    - Application is not caching any data and always makes calls to Cosmos DB
+
+- Reinforce the importance of using existing metrics/usage patterns/data to inform the load testing on their own application.

054-AzureLoadTesting/Coach/Solution-02.md

@@ -0,0 +1,6 @@
+# Challenge 02 - Create a Load Testing Script - Coach's Guide 
+
+[< Previous Solution](./Solution-01.md) - **[Home](./README.md)** - [Next Solution >](./Solution-03.md)
+## Notes & Guidance
+
+- Sample solution JMeter script is located in the solution directory [here](./Solutions/Challenge2/).

054-AzureLoadTesting/Coach/Solution-03.md

@@ -0,0 +1,36 @@
+# Challenge 03 - Create Azure Load Testing Service and Establish Baselines - Coach's Guide 
+
+[< Previous Solution](./Solution-02.md) - **[Home](./README.md)** - [Next Solution >](./Solution-04.md)
+
+## Notes & Guidance
+
+Below are the instructions to complete challenge 3
+
+- Create Azure Load Testing resource
+    - Search for the Azure Load Testing resource on the top search bar and select the resource.
+    - On the top left hand corner select "Create"
+    - Select the Resource Group that was created by the sample app or create your own.  The resources must be in the same location.
+    - Enter a unique name for the resource.
+    - Ensure the location matches the location that your application is deployed to.
+    - Select "Review + create"
+    - Review your configurations and then select "Create"
+- Create Load Test
+    - Go into your new resource.  You may see a warning on the top depending on your current access describing the access levels required.  If so, add yourself as one of the roles such as "Load Testing Owner" or "Load Testing Contributor" as you will not be able to proceed otherwise.
+    - Select "Create"
+    - Fill out the name of the test and a description of the test.
+    - Go to the "Test Plan" tab and select the JMeter script you have locally and then select "Upload"
+    - Under the "Parameters" tab, enter "webapp" for the name of the environment variable, followed by the URL for your sample app in the "Value" section.
+    - Under the "Monitoring" tab, select "Add/Modify" and select the resources from the sample app followed by "Apply" so they are monitored.
+    - Select "Review + create"
+    - Review the settings and select "Create"
+- Run the load test
+    - In your Azure Load Test service, select "Tests" from the left hand side
+    - Select the test that you created from the list
+    - At the top of the screen select "Run"
+    - A summary page should appear.  Add that this run is the baseline in the description.  Select "Run"
+- Set Pass/Fail Criteria
+    - Go back to your load test and select "Configure" followed by "Test"
+    - Go to the "Test Criteria" Tab.
+    - Enter in your test criteria such as failure rate under x %
+
+Sample solution JMeter script is located in the solution directory [here](./Solutions/Challenge3/).

054-AzureLoadTesting/Coach/Solution-04.md

@@ -0,0 +1,14 @@
+# Challenge 04 - Enable Automated Load Testing (CI/CD) - Coach's Guide 
+
+[< Previous Solution](./Solution-03.md) - **[Home](./README.md)** - [Next Solution >](./Solution-05.md)
+
+## Notes & Guidance
+
+
+High level Notes
+
+- There is a sample action which will run a load test.  The default action will either create or run the existing load test.  However, the "TestName" parameter inside the config.yaml file will be automatically lower cased by the GitHub Action.  Since the UI allows upper or lowercase letters, this can cause a duplicate test to be created.
+- GitHub will require another action to create issues while ADO has a built-in functionality to do this.
+- Students can use the baseline from their first run in challenge 3 to determine what their pass/fail criteria is.
+
+Sample solution JMeter script and configuration is located in the solution directory [here](./Solutions/Challenge4/).

054-AzureLoadTesting/Coach/Solution-05.md

@@ -0,0 +1,13 @@
+# Challenge 05 - Identify & Remediate Bottlenecks - Coach's Guide 
+
+[< Previous Solution](./Solution-04.md) - **[Home](./README.md)** - [Next Solution >](./Solution-06.md)
+
+## Notes & Guidance
+- This link goes into the issues in detail
+https://docs.microsoft.com/en-us/azure/load-testing/tutorial-identify-bottlenecks-azure-portal
+
+- The students should be seeing many errors and that Cosmos DB RU are maxed out.  If they are not, have them increase the amount of load in their tests.
+- Make sure the students have configured monitoring integration from Azure Load Test with the services such as App Service, App Insights, and Cosmos DB.
+    https://docs.microsoft.com/en-us/azure/load-testing/how-to-appservice-insights
+- Cosmos DB can either be scaled up manually or set to auto-scale.
+- Post scale tests should show a drastic decrease in errors and Cosmos DB RU nowhere near the max. You may still see some errors which was normal.

054-AzureLoadTesting/Coach/Solution-06.md

@@ -0,0 +1,9 @@
+# Challenge 06 - Stress Testing - Coach's Guide 
+
+[< Previous Solution](./Solution-05.md) - **[Home](./README.md)** - [Next Solution >](./Solution-07.md)
+
+## Notes & Guidance
+
+I start noticing failures when we reach 6 users.  It seems like 4 users will hit 60% of Cosmos DB consumption assuming a 1k limit.  There is then a sharp jump when you hit 10 users in terms of the number of failures.  Cosmos DB is still the bottleneck as our App Service is still running on one instance and does not appear to be constrained at all.
+
+Sample solution JMeter script is located in the solution directory [here](./Solutions/Challenge6/).

054-AzureLoadTesting/Coach/Solution-07.md

@@ -0,0 +1,37 @@
+# Challenge 07 - Load Testing With Chaos Experiment (Resilience Testing) - Coach's Guide 
+
+[< Previous Solution](./Solution-06.md) - **[Home](./README.md)**
+
+## Notes & Guidance
+
+Here are instructions on how to setup Chaos Studio and run the experiment.  Chaos Studio does not offer a GitHub Action at this time.  We want the users to get a taste of chaos engineering so they can trigger the chaos experiment manually while the load test runs.  They are not required to implement this all in the pipeline.  That is an advanced challenge.
+
+- Azure Chaos studio only supports a number of faults such as VMs, AKS, Cosmos DB, and networking to name a few.  Since our sample app only contains App Services and Cosmos DB, we will use the Cosmos DB faults.
+- First you will need to enable geo-redundancy in your Cosmos DB.  Go to your Cosmos DB instance and select "Enable Geo Redundancy" on the top bar.  Keep note of the paired region your Cosmos DB is paired with.
+- Create Azure Chaos Studio/Experiment
+    - Register Chaos Studio Provider
+        - Go to your subscription
+        - On the left-hand side, select "Resource provider"
+        - In the list of providers, search for "Microsoft.Chaos"
+        - Click on the provider and select "Register"
+    - Go to Azure Chaos Studio
+    - Under "Targets" select "Enable Targets" then "Enable service-direct targets"
+    - Select the Cosmos DB service for your load test.
+    - On the left-hand side, select "Experiments"
+    - Select "Add an experiment"
+    - Fill in your subscription, resource group, location, and name for this experiment.  Keep track of the experiment name as a managed user will be created for you.
+    - Go to the experiment designer on the next tab.  Change the name of the step or branch if you wish.
+    - Select "Add Action" follows by "Add Fault"
+    - Select "Cosmos DB Failover" as the Fault.  You can choose the duration but 2-3 mins should be enough.  Finally, enter the region that your instance is paired with in the "readRegion" section.
+    - Save your experiment.
+- Update Cosmos DB Permissions
+    - In Cosmos DB select "Access Control"
+    - Select "Add" followed by "Add Role Assignment"
+    - Select the CosmosDBOperator role, then click "Select members".
+    - Search the name of the experiment from the earlier step and then click "Select".
+    - Review and assign the permissions which will grant the role to the experiment.
+
+- Run load test + experiment
+    - Run the load test first, then while the load test is running kick off the chaos experiment.  You should notice that the application does not fail, but there will be a large spike in response time when you kick off the chaos experiment.  You should only see one spike and the app should return to normal behavior afterwards.
+
+