Skip to content

Commit

Permalink
changed content to conform to Content Guide
Browse files Browse the repository at this point in the history
  • Loading branch information
ehaines1 committed Nov 13, 2024
1 parent cef9892 commit 2333a8c
Show file tree
Hide file tree
Showing 3 changed files with 24 additions and 21 deletions.
14 changes: 7 additions & 7 deletions Testing/Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ ScubaGear repository consists of an automation suite to help test the functional

This README outlines the ScubaGear software test automation and its usage. The document also contains instructions for adding new functional tests to existing automation suite.

## Table of Contents <!-- omit in toc -->
## Table of Contents <!-- omit in toc -->

- [Functional Testing Prerequisites](#functional-testing-prerequisites)
- [Windows System with Chrome](#windows-system-with-chrome)
Expand All @@ -22,10 +22,10 @@ This README outlines the ScubaGear software test automation and its usage. The d
- [Nightly Functional Tests](#nightly-functional-tests)
- [Troubleshooting](#troubleshooting)
- [Issues with installing Pester](#issues-with-installing-pester)
- [Iusses with Selenium](#iusses-with-selenium)
- [Issues with Selenium](#issues-with-selenium)
- [Chrome browser version issue](#chrome-browser-version-issue)
- [Service principal authentication issue](#service-principal-authentication-issue)
- [Additional resources for admins](#additional-resources-for-admins)
- [Setup Documentation for System Admins](#setup-documentation-for-system-admins)

## Functional Testing Prerequisites ##

Expand Down Expand Up @@ -371,7 +371,7 @@ To Resolve above issue, try the following:
```
- Best practice is to install Pester and other modules as a non-admin user with either 'Current User' or 'AllUsers' scope. When modules are installed with 'AllUsers' scope, they are installed to the ``` $env:ProgramFiles\PowerShell\Modules``` location. The 'CurrentUser' scope installs modules to ``` $HOME\Documents\PowerShell\Modules``` location.

### Iusses with Selenium
### Issues with Selenium
We have seen issues with Selenium where test orchestrator would not run and produce random errors. If you have seen issues where error messages does not match with any other items in this troubleshooting section, try uninstalling and re-installing Selenium using following commands:
```
uninstall-module Selenium
Expand Down Expand Up @@ -416,9 +416,9 @@ If you are trying to run the test orchestrator as a service principal and your c

![service-principal-error](/images/service-principal.png)

### Setup Documentation for Sys Admins
The following resources are for M365 tenant admins to provide additional information on setting up the infrastructure (service principals, user provisioning, etc.) for functional testing of ScubaGear.
### Setup Documentation for System Admins ###
The following resources are for M365 tenant admins to provide additional information on setting up the infrastructure (service principals, user provisioning, etc.) for functional testing of ScubaGear.

- [How to setup the permissions required to execute the automated functional test orchestrator](./docs/SetupFunctionalTestOrchestratorServicePrincipalAndAssociatedPermissions.md)

- [How to setup a tenant with the necessary AAD conditional access policies to run the Automated Functional Test Orchestrator](./docs/SetupTenantForTestingAADConditionalAccessPolicies.md)
- [How to setup a tenant with the necessary AAD conditional access policies to run the Automated Functional Test Orchestrator](./docs/SetupTenantForTestingAADConditionalAccessPolicies.md)
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
### Setup the Functional Test Orchestrator service principal and associated permissions
# Setup Functional Test Orchestrator service principal

This section describes how to setup an AAD application in the tenant when you want to run the test orchestrator using a **service principal (non interactive login)**. Setup for user interactive login is documented in a separate section.

Go to Azure AD > App Registrations and click New registration
Expand All @@ -16,7 +17,8 @@ Click Add a permission then select Microsoft Graph in the popup page. Note some

Select Application permissions and then add all of the required permissions in the list below which are required for AAD. Once you have selected all of the permissions, click the Add permissions button.

#### Microsoft Graph API permissions
## Microsoft Graph API permissions

- Directory.Read.All
- GroupMember.Read.All
- Organization.Read.All
Expand All @@ -27,10 +29,11 @@ Select Application permissions and then add all of the required permissions in t
- PrivilegedAccess.Read.AzureADGroup
- RoleManagementPolicy.Read.AzureADGroup

#### Office 365 Exchange Online API permissions (select from APIs my organization users)
## Office 365 Exchange Online API permissions (select from APIs my organization users)

- Exchange.ManageAsApp (so the application can run cmdlets in Exchange Online)

#### Sharepoint API permissions (select from Microsoft APIs)
## Sharepoint API permissions (select from Microsoft APIs)

- Sites.FullControl.All (so the application can update Sharepoint settings)

Expand All @@ -47,10 +50,11 @@ Click the Grant admin consent button on the API permissions page and click Yes i
The permissions page should now show that admin consent was granted for each of the permissions
![image](https://github.com/cisagov/ScubaGear/assets/107076927/6065fcba-f3c3-4a37-944f-f19c4c7e0e7a)

### Assigning user roles to the Scuba Functional Test Orchestrator application
## Assigning user roles to the Scuba Functional Test Orchestrator application

Some of the products also need an AAD user role assigned to the application in order for it to be able to update the tenant settings when executing the functional tests.

#### Assign the following user roles to the "Scuba Functional Test Orchestrator" application using the AAD role assignments page. Note that these should be active assignments in tenants that include PIM.
## Assign the following user roles to the "Scuba Functional Test Orchestrator" application using the AAD role assignments page. Note that these should be active assignments in tenants that include PIM

- Exchange Administrator (for EXO and most of the Defender cmdlets except for the compliance ones)
- Compliance Data Administrator (for Defender since it uses Purview compliance center cmdlets such as Set-DlpCompliancePolicy, Set-DlpComplianceRule, Set-ProtectionAlert)
Expand All @@ -62,8 +66,6 @@ Here is an example screenshot that shows the service principal assigned to the E

To complete the setup for PowerPlatform you must also execute the code below to register the service principal with Power Platform:

```
``` PowerShell
Add-PowerAppsAccount -Endpoint prod -TenantID $tenantId # use -Endpoint usgov for gcc tenants
New-PowerAppManagementApp -ApplicationId $appId
```
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
### Setup the tenant for testing AAD conditional access policies in the functional test plan
# Setup the tenant for functional testing of AAD conditional access policies

Numerous test cases in the AAD functional test plan associated with conditional access rely on some dependencies in the tenant that you must setup ahead of time:

- **Step 1** - Create a conditional access policy in the tenant named **Automated Test 1 - DO NOT MODIFY** and set it up as per the configuration in the screenshot below. Set the policy to **Report-only** (do NOT turn it on). This policy is downloaded from the tenant when the test orchestrator executes and its characteristics are modified in memory (using the test orchestrator's RunCached function) before the provider JSON is sent to the Rego. This CA policy must be configured exactly as shown below in order for the test cases to work correctly. The only purpose of this CA policy is to support the execution of the test plan - the policy serves no purpose to the actual operations of the tenant and hence why it must be set to Report-only.

![image](https://github.com/cisagov/ScubaGear/assets/107076927/ae19af14-c551-4ef1-b98b-e2e32087bcb8)

- **Step 2** - The following conditional access policies are also required to be created in the tenant and configure as per the respective AAD baseline policy instructions, including setting Enable Policy to **On**. The policy names in the tenant must be configure exactly as described below:
- MS.AAD.1.1v1 Legacy authentication SHALL be blocked
- MS.AAD.2.1v1 Users detected as high risk SHALL be blocked
- MS.AAD.2.3v1 Sign-ins detected as high risk SHALL be blocked
- MS.AAD.3.2v1 If phishing-resistant MFA has not been enforced, an alternative MFA method SHALL be enforced for all users
- MS.AAD.1.1v1 Legacy authentication SHALL be blocked
- MS.AAD.2.1v1 Users detected as high risk SHALL be blocked
- MS.AAD.2.3v1 Sign-ins detected as high risk SHALL be blocked
- MS.AAD.3.2v1 If phishing-resistant MFA has not been enforced, an alternative MFA method SHALL be enforced for all users

0 comments on commit 2333a8c

Please sign in to comment.