merge all lme 2.0 changes into release-2.0.0
## 🗣 Description ##

<!-- Describe the "what" of your changes in detail. -->

Add dashboard-descriptions.md in /docs/markdown/reference.
Add a link to this file within the main README.md's table of contents.

### 💭 Motivation and context 

<!-- Why is this change required? -->
<!-- What problem does this change solve? How did you solve it? -->
<!-- Mention any related issue(s) here using appropriate keywords such -->
<!-- as "closes" or "resolves" to auto-close them on merge. -->

The LME repository does not have a location for dashboard descriptions.

## 🧪 Testing 

<!-- How did you test your changes? How could someone else test this PR? -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->

N/A

## ✅ Pre-approval checklist ##

- [x] Changes are limited to a single goal **AND** 
      the title reflects this in a clear human readable format
- [x] I have read and agree to LME's [CONTRIBUTING.md](https://github.com/cisagov/LME/CONTRIBUTING.md) document.
- [x] The PR adheres to LME's requirements in [RELEASES.md](https://github.com/cisagov/LME/RELEASES.md#steps-to-submit-a-PR)
- [x] These code changes follow [cisagov code standards](https://github.com/cisagov/development-guide).
- [x] All relevant repo and/or project documentation has been updated to reflect the changes in this PR.

## ✅ Post-merge Checklist

- [x] Squash all commits into one PR level commit 
- [x] Delete the branch to keep down number of branches

* Update README.md to include dashboard-descriptions.md

* Update wording for computer software overview dashboard

* Fix some grammatical changes in dashboard-descriptions.md

* Release 1.3.1 merge into main  (#154)

* Update retention function to fix retention policy bug (#143)

* Updated troubleshooting guide to account for index management (#134)

* Update upgrading.md to account for 1.3.1 (#151)

* Update upgrading.md

* Update upgrading.md

---------

Co-authored-by: Andrew Arz <[email protected]>

* Fixes dashboard update not importing on fresh install (#167) (#169)

* Fixes dashboard update not importing on fresh install
#165

* Update upgrading.md to include status on v1.3.2, along with revisions to the document overall

* remove step 4 from upgrading.md; add additional instructions for v1.3.2

---------

Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Clint Baxley <[email protected]>

* Add proof of concept selenium tests

* Correct the script name in the doc string

* User Security Selenium Tests for No Results Panels

* First full selenium test. Currently just User Security

* WIP User HR

* Completed all dashboards. Requires testing now

* Cut dev comments

Co-authored-by: Alden Hilton <[email protected]>

* Debugging a couple unit tests that error out. Two left

* Install LME in the testbed from a single script (#150)

* Adding the configure scripts

* Add scripts to zip and copy to a container for downloading in the server

* Grab the expiry time properly in copy file

* Overwrite the blob if it exists

* Add the script to download file into DC

* Script that unzips the files in a container

* Adds username argument to download files

* Add script to run scripts in container

* Adds username argument to gpo script

* Modifies the url name in the client GPO

* Adds the functionality for chapter 1 and first half of chapter 2

* Imports the sysmon GPO

* Update the variables for sysmon gpo

* Name the scripts so they are grouped together in a listing

* Echos the file download url

* Expands the domain name correctly in create ou

* Write the url output of copy file to container to a different output stream

* Create a new LME folder for our scripts and files

* Set path for extract to lme

* Update paths for scripts to /lme

* Fix the wec server name setting

* Adds the scripts to install chapter 1 and 2

* Allows azure to download in linux and windows

* Adds linux install scripts.

* Adds winlogbeat installer

* emove garbage in update server name

* Tweak several scripts to get the scp of files_for_windows

* Adds installer script to run all the scripts

* Fixes the formatting method for az output

* Clean up the scripts and add documentation

* Fixes outputting format errors

* Fixes hanging on adding ls1 to domain

* Fix formatting errors on responses

* Update linux expect script for different prompts.

* Handle the reboot message for linux expect script

* Echos the file download url

* Create a new LME folder for our scripts and files

* Set path for extract to lme

* Update paths for scripts to /lme

* Update paths for scripts to /lme

* Fix the wec server name setting

* Adds the scripts to install chapter 1 and 2

* Allows azure to download in linux and windows

* Adds linux install scripts.

* Adds winlogbeat installer

* emove garbage in update server name

* Tweak several scripts to get the scp of files_for_windows

* Adds installer script to run all of the scripts

* Fixes the formatting method for az output

* Clean up the scripts and add documentation

* Fixes outputting format errors

* Fixes hanging on adding ls1 to domain

* Fix formatting errors on responses

* Update linux expect script for different prompts.

* Handle the reboot message for linux expect script

* Adds InstallTestbed instructions to Readme.md

* Modifies parameters to be pascal case

* ls1 not being set on DC1

* Adds Linux Only install to SetupTestbed

* Remove separate linux only script

* Update testing/Readme.md

Co-authored-by: Alden Hilton <[email protected]>

* Make number of clients consisten between scripts

* Add ports for elk stack for testing

* Update readmes to change ResourceGroupName to ResourceGroup

* Adds a switch to install linux only

* Adds simple tests to check install

* Removes the error if the old configure zip is not found.

* Adds variables to linux tests run command

* Move credential extraction to lib for use by other scripts.

* Adds npm for other testing

* Adds latest version of nodejs for testing

* Make output.log readable for tests

* Add the -m parameter in the testing readme

* Download the latest version or a specified version

* Reboot for 1.3.0

* Notes that we could have different expect scripts

* Put back in the restart after all of the domain updates

* Scp uses ls1 instead of ls1.lme.local

* Up the timeout of the adding ls1.lme.local

* Up the timeout of the adding ls1.lme.local

* Fixes chmod of the output.log for tests

* Adds venv to the gitignore

* Adds the ability to pass a branch to the installer

* Remove node installer

* Change timeout in expect script for slow connections

* Make shell files executable

---------

Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Alden Hilton <[email protected]>

* Fix deploy.sh data retention failure error  (#190)

* Fix deploysh data retention failure (#179)

* Update deploy.sh

* Update deploy.sh

* Update deploy.sh

* Update deploy.sh

* Remove free (#188)

* changed the word free to no-cost or no-cost to users

* rephrased wording to 'which comes at no cost to users'

---------

Co-authored-by: Linda Lovero-Waterhouse <[email protected]>

* Update upgrading.md with data retention failure resolution (#189)

---------

Co-authored-by: Andrew Arz <[email protected]>
Co-authored-by: Linda Waterhouse <[email protected]>
Co-authored-by: Linda Lovero-Waterhouse <[email protected]>

* Automatically Add Tags to Azure Resources (#186)

* Add tags to all Azure resource creations calls

---------

Co-authored-by: Clint Baxley <[email protected]>

* Switched script to headless mode

* added switch for headless, detached, and debug mode. Bug where driver.quit does not close window.

* Refactored long line and added switch for debug mode

* Removed unnecessary comments

* Update pull_request_template.md (#198)

* Update pull_request_template.md

Moved Squash commits from post-merge to pre-merge.

* overriding default PR template for preferred LME template

* overriding default PR template for preferred LME template

* updating issue template to shorten the template

---------

Co-authored-by: mreeve-snl <[email protected]>

* Python testbed setup (#183)

* Add simple tests for http requests

* Add an env file to gitignore

* Remove unneeded pip install

* Hide pytest_cache

* Add pycache to gitignore

* Adds dev containers for vscode

* Adds testing information for vscode

* Uses .env file for tests if present

* Adds env example file

* Modify development container name

* Adds readme for the testing environment

* Add simple tests for http requests

* Add an env file to gitignore

* Remove unneeded pip install

* Adds dev containers for vscode

* Adds testing information for vscode

* Uses .env file for tests if present

* Adds env example file

* Modify development container name

* Adds readme for the testing environment

* Create helpers and conftest for python tests

* Setup for using test explorer in the dev environment

* Adding azure shell requirements to docker image

* Adding Python API tests

* Merges additional tests

* Made changes to fix tests that were failing

* Separate linux only tests from others

* Create a workflow for building test environments

* Make the docker user be the same as the vbox user id

* Set up to run the installer in docker

* Pick up different fs types in data_retention

* Change the build path for building lme container

* Install lme after build

* Make lme installer executable

* Set up the build for tests

* Add the cluster workflow for github actions

---------

Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Rishi <[email protected]>

* Update PULL_REQUEST_TEMPLATE.md (#206)

Added instruction to select Issue in Development area so that the corresponding Issue is automatically closed when the PR is merged.

* Made changes to facilitate HTML Reports on test execution  (#211)

* Made changes to requirements.txt, ReadMe and gitignore to facilitate HTML reporting

* Fixed Typos on Readme

* Fixed Typos on Readme

* removed tags flag from nsg because it was preventing some rules from being created (#214)

Co-authored-by: Linda Lovero-Waterhouse <[email protected]>

* Update PULL_REQUEST_TEMPLATE.md (#217)

Using keywords like "fixes" or "closes"  only auto-closes the corresponding issue if the PR is going to be merged into main.  

For PR's merged into release branches, we need to add the issue to the development box in the right sidebar in order to auto-close the issue.  

Added some documentation to clarify this.

* Create new workflow for automating the release process (#199)

* Github workflows for building environments (#195)

* Run the correct installer file

* Run the installer from the root directory

* Try a self hosted github runner

* Reduce logging for docker pull.

* Adds quiet flag to docker pull command

* Pull the images before expect to reduce run time

* Install docker early in order to speed up install

* Builds the right docker-compose file

* Increase timeout for linux install expect script

* Change timeout on expect script

* Change the way expect watches the script

* Expand the timeout when waiting for Elasticsearch

* Search for more output in the expect script

* Change the match for the dots in expect

* Change the regex for matching dots

* Change the output for catching dots

* Add chrome to Dockerfile for selenium

* Import selenium tests and run python tests

* Activate venv when running tests

* Correct path for venv in the container

* Correct path for venv in the container

* Running only linux tests

* Adjust scripts to run as a non super user

* Change the permissions on the output log to source for environment variables later

* Check for output log

* Make output log available to test instantiation

* Change pytest cache dir to home for user

* Change pytest cache dir to home for user

* Change pytest cache dir permissions

* Hide get-docker.sh from installs

* Cleanup test files in workflow

* Add the cluster workflow for github actions

* Adds a cluster build

* Run the test cluster in pwsh

* Fail pipeline when commands fail

* Catch the error from powershell

* Remove duplicate run command

* Set env vars explicitly

* Modify the escape char for env vars

* Try a different method of catching errors in pwsh script

* Check failure of pwsh script

* Test successful run of build_cluster

* Test failure of script

* Capture the output from the az commands

* Continue on error condition

* Simplify run command

* Try catching failures in a new way.

* Test failure capture

* Setting error action to continue

* Remove ErrorAction

* Use docker-compose run instead

* Capture exit code to fail step

* Try propigating errors from pwsh

* Capture external command exit code

* Send lastexitcode

* Don't exit right away

* Disable immediate stop on exit

* Run simple test for exit code

* Cd to docker compose file

* Catch exec exit code

* Remove unneded flags from the command

* Adds back in the build script

* Adds an explicit exit for powershell script

* Remove spaces after escape character

* Escape the exitcode variable in the shell command

* Remove extra exit from build_cluster.ps1

* Add a passing command for build_cluster.ps1

* Move to the install directory

* Run setup testbed to get an error

* Try to build a cluster with the build_cluster.ps1 script.

* Check resource group variable

* Set the resource group name differently

* Build a cluster using the generated resource group

* Make the paths relative in the build_cluster script

* Move to the right directory to do an install

* Destroy cluster on pipeline finish

* Change the owner of the files to match the host in the development container

* Su user to remove testing files

* Run the docker-compose as root to clean up

* Run as root to clean up containers

* Build the cluster in azure

* List the files in the current directory on exec

* Run the files from the new path

* Investigate more about the file environment

* Update the envornment for building the cluster

* Update the environment users before docker up

* Try to start hung job

* List all the files with their owners in the container

* Escape the powershell commands

* Check the paths and files with bash

* Find the path we are on

* Check powershell environment

* Cd to home directory in powershell

* Cd to home directory in powershell

* Rebuild docker compose as the right user

* Change directory to source directory for powershell

* Change to proper directory for powershell

* Build a full cluster in pipeline

* Run the linux tests and check permissions of files

* Change permissions on output file with sudo

* Turn off cluster creation for speed

* Comment out building cluster in steps

* Only delete the resource group if it exists

* Adds ability to get the public ip for fw rules

* Put the tags in quotes when creating nsg rules

* Output the command being run for nsg rules

* Remove tags for nsg port definitions

* Install lme on the cluster

* Builds the full cluster install

* Cleans up the useage of the environment variables in pipeline

* Extract environment variables from the build script and use them in the GitHub workflow.

* Do a minimal linux install

* Fix the path for retrieving env vars

* Check setting of github env

* Source the env file and push it to github env

* Print some debug information to the console

* Check setting of each key in functions

* Parse the output for the passwords better

* Uses a unique id instead of run_id to make sure it is unique

* Double quote the file name for sed in output.log

* Changes the way we get passwords from output.log

* Make sure key doesn't have newline

* Escape dollar sign

* Properly escape double quotes inside of docker-compose command

* Escape all of the dollar signs in the compose command

* Write the environment variables to the githut environment

* Clean up debugging output

* Remove more debugging output

* Remove set e

* Adds function to write passwords to a file for actions

* List files in directory after writing passwords

* Export the env vars in the github file

* Fail the workflow if the environment is not set correctly

* Clean up the environment vars for the container

* Set the variables on run of the pwsh command

* Run commands on the domain controller

* Get the envrionment checker to pass

* Update passing variables to remote script

* Escape the powershell environment variables

* Change the case of the resource group env var

* Don't destroy cluster so we can manually test

* Build the entire cluster to run commands against

* Run a command on the linux machine

* Run remote tests

* Run minimal installs to debug tests

* Fix escaping for test commands

* Move to the correct directory for tests

* Add continuation characters to the lines in the script

* Remove nested double quotes

* Uses the ip of LS1 to run the tests on

* Put the cluster build command on one line

* Destroy clusters at the end

* Quote output log correctly on build

* Run all api tests on cluster

* Build full cluster and add verbose logging to pytest

* Stop deleting the cluster in the destroy_cluster.ps1 script

* Modify installer to use the new winlogbeat index pattern

* Try to get the dns to resolve ls1

* Add ls1 to the hosts file so it resolves always

* Modify tests to pass on a working cluster

* Skip the fragile test for mapping

* Set up to run selenium tests on the cluster

* Testing

* Rerun build after rebasing to the right branch

* Pass the minimal install flag to install lme

* Build complete cluster and run all tests

* Pull the images quietly if running without a terminal.

* Run the simple tests on PR checkin and the longer ones when triggered

* Build the linux docker container upon check in of a pr

* Build lme container fresh before install

* Runs an end to end build in docker and cluster

* Print out the download feedback when pulling images

* Build 1.4.0 branch

* Build the cluster using the main branch of the repository

* Allow passing branch to installers from the pipeline

* Run tests from a different base branch

* Remove the ampersand typo

* Allow passing arguments to the installer scripts

* Rearrange install arguments

* Test passing arguments in install lme

* Build lme without arguments

* Install lme with no arguments

* Run command as string in install_lme.ps1

* Build by passing arguments

* Run a complete build using arguments

* Update the sources to allow for updating in the pipeline

* Build the cluster using the latest branch

* Set up the latest branch var

* Runs an upgrade in the pipeline

* Run the upgrade in the remote linux machine

* Run upgrade on minimal install

* Checks out the current branch to run an upgrade on linux

* Capture the exit code of the upgrade script

* Check the directories we are working in

* Clone the git repository to run the upgrade

* Checkout the proper branch from origin

* Get the remote username and home dir for the remote server

* Set the home directory for the az user

* Use origin when checking out in the upgrade script

* Revert the changes to deploy.sh

* Set a dumb terminal to avoid terminal errors

* Export the terminal variable correctly

* Capture the output of the upgrade script to fail pipeline if it fails

* Revert previous changes as they seemed to break upgrade

* Use a different format for executing the pwsh script

* Destroy the cluster when done

* Output the upgrade information to the terminal

* Try capturing the docker-compose output

* Directly capture the output of the compose command

* Fixes unbalanced quote

* Build and run full cluster with an upgrade

* Builds the current brand for the cluster

* Add a unique id for the docker-compose so you can run multiple instances of the same docker-compose file

* Adds upgrade.yml to gh workflows

* Runs both a build and an upgrade

* Adds upgrade to the gh workflows

* Get gh to notice new workflow

* Match build names to parent branch

* Trigger gh to see the workflow

* Get gh actions to trigger workflow

* Update code to get gh to see the actions

* Update code to use the new workflow module.

* Trigger gh actions to run

* Get gh to run workflows

* Try to get gh to run workflows

* Change upgrade branch pulling

* Checking out branch for upgrade in a new way

* Rename workflow for upgrade

* Convert to docker compose

* Run all three builds using docker compose and -p

* Clean up docker containers

* Build the docker containers fresh for the linux_only workflow

* Adds readme and checks an upgrade where the upgrade version is the same as the current version

* Fixes typo in the workflow file

* Runs docker as sudo

* Remove the privileged flag from the lme container

* Try leaving the swarm on the host if running in non privileged environment

* Leave the swarm on the host

* Reset to run docker as privileged

* Installs the current branch in linux only

* Stop pruning system to see if elastc starts faster

* Don't take down the docker containers to see why they aren't working

* Removes the gh actions shell escape vulnerability

* Remove the docker containers at end of run

* changing .github/README.md name to prevent it apperaing on main web page (#260)

* Append the flags to the end of the password file (#263)

* Append the flags to the end of the password file

* Prints the contents of password.txt to the console

* Extract the credentials in a new way to compensate for the flags being in the file

* Tests a build that runs locally on github

* Keep container running for debugging purposes

* Fix the credentials parsing function

* Create a workflow for a burndown chart (#302)

* Display the chart in the burndown summary
* Get workflow dispatch to show
* Adds defaults for the burndown chart workflow

* Clean up debugging information from the workflow (#310)

* Clean up debugging information from the workflow

* Increase column count to match the number of columns in the board.

* Break up selenium tests (#281)

* Adding selenium directory and readme

* Separate out the selenium tests so they can be run separately

* Run selenium tests in pipeline

* Puts the variables for env one to a line

* Issue # 289 selenium test for Computer Software Overview  dashboard (#290)

* Updated Selenium tests for Computer Overview Dashboard

* Updated Selenium tests for Computer Overview Dashboard

* Updated Selenium test scripts for Health Check Dashboard (#292)

* Set up selenium tests to run on cluster test

* Point tests to the proper test folder

* Update Selenium tests for Process Explorer Dashboard (#295)

* Rewrite completed for Selenium test scripts for Security Dashboard - Security Log (#300)

* Rewrote Selenium Tests for Sysmon Summary Dashboard (#301)

* Rewrite Selenium Tests for User HR Dashboard

* Rewrite of Selenium Tests for User Security Dashboard (#304)

---------

Co-authored-by: rishagg01 <[email protected]>
Co-authored-by: Rishi <[email protected]>

* API calls code for Data Insertion (#343)

* modified:   testing/tests/api_tests/helpers.py
	new file:   testing/tests/api_tests/selenium_tests/__init__.py
	new file:   testing/tests/api_tests/selenium_tests/conftest.py
	new file:   testing/tests/api_tests/selenium_tests/fixtures/hosts.json
	new file:   testing/tests/api_tests/selenium_tests/fixtures/logonevents.json
	new file:   testing/tests/api_tests/selenium_tests/queries/filter_hosts.json
	new file:   testing/tests/api_tests/selenium_tests/queries/filter_logonevents.json
	new file:   testing/tests/api_tests/selenium_tests/test_server.py

* commit	renamed:    testing/tests/api_tests/selenium_tests/__init__.py -> testing/tests/api_tests/data_insertion_tests/__init__.py
commit	renamed:    testing/tests/api_tests/selenium_tests/conftest.py -> testing/tests/api_tests/data_insertion_tests/conftest.py
commit	renamed:    testing/tests/api_tests/selenium_tests/fixtures/hosts.json -> testing/tests/api_tests/data_insertion_tests/fixtures/hosts.json
commit	renamed:    testing/tests/api_tests/selenium_tests/fixtures/logonevents.json -> testing/tests/api_tests/data_insertion_tests/fixtures/logonevents.json
commit	renamed:    testing/tests/api_tests/selenium_tests/queries/filter_hosts.json -> testing/tests/api_tests/data_insertion_tests/queries/filter_hosts.json
commit	renamed:    testing/tests/api_tests/selenium_tests/queries/filter_logonevents.json -> testing/tests/api_tests/data_insertion_tests/queries/filter_logonevents.json
commit	renamed:    testing/tests/api_tests/selenium_tests/test_server.py -> testing/tests/api_tests/data_insertion_tests/test_server.py
commit	modified:   testing/tests/api_tests/helpers.py

* Updated selenium tests for USER HR dashboard panels post data insertion (#358)

* adding ignore for vim files

* moving old readme to old_chapters directory

* moving chapters to old_chapters folder

* Committing Readme changes and updates and removing old backups directory

* Adding Configuration files for lme 2.0

* Adding Ansible Playbook Yaml for installing lme 2.0

* Committing Quadlet files for LME 2.0 arch

* Adding Scripts:

- download.sh/upload.sh: upload/download logs in mass from elasticsearch
  (will be integrated into future merging from 1 -> 2)
- link_latest_podman_quadlet.sh: links from the nix store the latest
  podman version into its expected directories
- set-fleet.sh: sets up the required fleet settings on kibana
- set_sysctl_limits.sh: sets the sysctl_limits as required by the
  architecture and containers
- install_lme_local.yml: sets up the ansible playbook for lme 2.0
  installation.

* move lme playbook to scripts directory

* pushing some more documentation to Readme

* initial diagram

* pushing updates to Readme to document ports/services/etc...

* Updated User HR Dashboard Selenium Test for User HR Logon Title panel (#385)

* Updated selenium tests for USER HR dashboard panels post data insertion

* Updated User HR Dashboard Selenium Test for User HR Logon Title panel

* Merge in the pipeline files

* Adds in the tesing installers

* Updates the paths to the LME install scripts

* Make the user create the environment file before doing install

* Make the lme-environment file so the install succeeds

* Adding pre-reqs to main testing/v2 readme

* Add some extra to the readme.

* Associate the nsg with the public ip

* Associate the nic instead of ip to the nsg

* Change default ports for nsg

* Update Caddyfile to include access log

* Adds back in some files from Chapter 3

---------

Co-authored-by: mitchelbaker-cisa <[email protected]>
Co-authored-by: Andrew Arz <[email protected]>
Co-authored-by: Clint Baxley <[email protected]>
Co-authored-by: Alden Hilton <[email protected]>
Co-authored-by: unknown <[email protected]>
Co-authored-by: Grant (SNL) <[email protected]>
Co-authored-by: Alden Hilton <[email protected]>
Co-authored-by: Linda Waterhouse <[email protected]>
Co-authored-by: Linda Lovero-Waterhouse <[email protected]>
Co-authored-by: Brown <[email protected]>
Co-authored-by: mreeve-snl <[email protected]>
Co-authored-by: Rishi <[email protected]>
Co-authored-by: rishagg01 <[email protected]>
Co-authored-by: Connor <[email protected]>

* Adds scripts to import and export 1.x data

* Modifies the import script to use podman

* Adds the dashboard importer for 1.x to 2.0

* Updates the import and export scripts to add mappings

* Updates the field limit on winlogbeat index upon import

* Moves the upgrade scripts to a folder and requires directory on import

* Adds ability to remove the old docker volumes

* Puts the volume remover in the upgrade directory

* Makes the volume remover executable

* 2x readme

* Increase default maximum field limit

* Alter title of imported dashboards to indicate 1x import

* Clarify some points in the upgrade readme

* Read the passwords and username from the config file if it exists

…ser Security Dashboard (#426)

* Updated API & Sel tests for Create Remote Threads panel

* Updated tests for Powershell network connections panel

* Updated APi & Sel test for Raw Access Read panel on User Security Dashboard

* Adds scripts to import and export 1.x data

* Modifies the import script to use podman

* Adds the dashboard importer for 1.x to 2.0

* Updates the import and export scripts to add mappings

* Updates the field limit on winlogbeat index upon import

* Moves the upgrade scripts to a folder and requires directory on import

* Adds ability to remove the old docker volumes

* Puts the volume remover in the upgrade directory

* Makes the volume remover executable

* 2x readme

* Increase default maximum field limit

* Alter title of imported dashboards to indicate 1x import

* Clarify some points in the upgrade readme

* Save this intermediary version of the docker files

* Updates docker and linux only workflow for 2.0

* Updates the paths for the linux only containers

* Fixes the clean up script for linux only build

* Get the logs from broken container

* Use root for the docker-compose.yml file

* Use azure for our installs on pipeline

* Installs python modules before azure install

* Pass azure env vars to the docker azure install script

* Updates the paths to the installer variable files

* Changes the paths for the environment vars for installer

* Change the password argument for the installer

* Comment out group removal for debugging

* Make sure the containers are using the same id

* Add the resource group prefix to the environment variables files

* Leave out special chars in password generation

* Put in a pause to wait for the linux machine to be ready

* Increase azure test machine size

* Speeds up pipeline docker creation

* Add sshpass to the apt packages in the Dockerfile

* Sleep after making ssh key

* Show output of generating key

* Adds the openssh-client to the doccker build

* Run the tests remotely

* Quote ssh commands and escape environment vars $

* Install chromium for tests

* Separate installing requirements from the test step

* Change default variables for tests

* Skip the tests that don't work with 2.0

* Clean up azure resources when pipeline is done

* Update the cluster build to use the new installers

* Update unique id and branch name

* Check permissions on folder for config files

* Rebuild container with correct uid

* Check if directories are writable

* Puts the env file in the proper directory

* Skips data insertion example tests

* Change the default password for selenium tests

* Skip selenium tests that point to old dashboards

* Skip failing tests

* Install minimega

* See if selenium tests pass without minimega

* Skipped failing test.

* Install linux in minimega

* Quote minimega arguments correctly

* Runs minimega as root

* Provide full path to minimega

* Runs minimega on the remote machine

* Remove the local call to minimega

* Get the azure and minimega ips in a variable for gh actions

* Better method to get the minimega IP

* Escape the arguments to getting the ip on minimega

* Attempt escaping again

* Get ip of the linux vm using lib function

* Updates development files and workflows for the pipeline

* Fail if the minimega ip isn't found

* Increase the size of the cluster azure instance

* Check if tests pass without minimega

* Install minimega first because it restarts machine

* Uses the machine name of a running vm

* Output the reason for not getting the minimega ip

* Escapes the azure ip $ sign

* Checking the ssh command

* Echo IP early

* Gets the ip for minimega and doesn't check errors

* Get the vm info for the vm in minimega

* Filter the ip outside of the remote command

* Filter the ip inside of the ssh command

* Use single quotes to quote the jq query

* Waits for an ip to be assigned to the minimega vm

* Get the policy and token for elastic agent

* Retrieve token after installing LME

* Wait for the services to come up before running set-fleet

* Put the check service command in the ssh command

* Run set fleet as sudo because it has podman available

* Source bashrc for podman path

* Try getting path to podman

* Echo path variable

* Check for podman path

* Put in absolute path to podman

* Remove install fleet

* Attempt running set fleet in the pipeline

* Fix the typo  in the pipeline docker build

* Turn on debug for set fleet

* Add a script to check the variables and results of set fleet

* Run the check fleet script before installing

* Update the fleet check script

* Print debug info from kibana

* Prints out the fleet api response.

* Waits for fleet to be ready

* Turn off debugging for the fleet installation scripts

* Take out some debugging and sleeps

* Run a command in a minimage virtual machine

* Ssh to the virtual machine using non root

* Use the env vars to connect to the ssh instances

* Ignore strict host checking in ssh

* Don't shut down instance so we can debug

* Test running sudo in minimega virtual machine

* Have pipeline ignore the certs when getting token and policy

* Use unique container names

* Try running in a different azure zone

* Updates the ip in the config file

* Fix the password for azure machine

* Sleep a little after azure machine creation

* Keeps azure resources in place after pipeline run

* Fix yaml error in workflow file

* Fix error in cluster.yml

* Echo enrollment token for debugging

* Repllace the vars in the config file for the local IP

* Copy the install_agent_linux.sh script to Minimega

* Test the install_agent_linux.sh script in Minimega

* Try running the Elastic Agent installer in Minimega

* Make the install_agent_linux.sh script executable and run it in Minimega

* Run the chmod and install_agent_linux.sh script in separate steps

* Run the agent installer with automatic "yes" response

* Quiet the untarring command

* Reduce logging for pulling the elastic agent

* Pass the enrollment token to the agent installer

* Try enrolling after installation

* Allow insecure enrollment

* Start the agent from /opt and restart the service after enrolling

* Run enroller non interactively

* Force enrollment

* Build the entire run again to test manually

* Checks if the elastic agent is reporting

* Sleep a little while waiting for results from agent

* Try to separate out installation, config, and enrollment of agent

* No need to run config. Enroll will do it

* Clean up the azure resources after the run

* Upgraded API & Sel tests over Rel 2 pipeline

* Updated API tests for Rel 2

* adding vim ignore to gitignore, so local files aren't included

* adding first working PoC

* NIST Guidelines (NIST Special Publication 800-63B): compliance

* lme environment will only be copied if it doesn't already exist to save old lme.environment

* Have all quadlets start as lmed user

* remove caddy from the architecture

* Make certs globally readable if volume is mounted

* Removing lmed, adding usernamespace, fixing volume permissions

* Commiting new quadlets:
  - run via administrator systemd
  - user UserNS to execute in new, unpriviliged userns
  - volumes so we don't run into permissions issues
  - add network/volumes to lme.service for restart

* Fix Ansible into multiple plays, change installation

* Finish up fixing ansible

* Fix up v2 testing docs

* Fix link latest podman bug

* Fix issue where permissions happen for elasticsearch data

* Fix some docs and push password manager

* Docs for security model AND architecture diagram

* Push up Table of Contents + Diagram into main readme

* Updating scripts with some comments and reorg

* Setting up ansible to create user passwords using ansible vault

* Pushing changes to quadlets

* Pushing changes to scripts

* Adding in docs changes + ansible script

* Updating ansible script to setup the passwords automatically

* Fix so nix is setup before settting up user acct passwords

* Pushing updated ansible script so that password can be set manually if desired

* Updating docs

* Don't delete the azure resource so it can be worked on

* Use a script to extract the secrets

* Don't need to remove the /tmp file, it has no secrets

* Extract the secrets all the way.

* Prints out debugging info for setting fleet

* More debugging info for setting fleet

* Don't comment the variables when setting fleet

* Escape backslashes in env file

* Set locale to en_US.UTF-8

* Run as root when setting fleet

* Debugging set-fleet.sh

* Delete Azure resources

* Change set-fleet.sh to use the secrets in the vault

* Show the secrets in the vault for debugging

* Update set-fleet.sh to output the response from the Fleet API

* Source the export instead of executing it

* Keep azure running to set fleet manually

* Replace more of the hardcoded secrets with env vars

* Sets elastic password as an env var for the pipeline

* Make the azure ip address available to the pipeline

* Remove newlines from the retrieve password command

* Get the azure ip address and make it available to the pipeline

* Don't comment out the Azure IP address

* Capture only the last line of the elastic password

* Put the elastic password in the .env file

* Hide the elastic password in the github actions logs

* Hide the elastic password in the github actions logs

* Hide the output of the .profile source

* Allow the .profile source to fail

* Add the kibana password to the github actions environment

* Add some more environment variables to the .env file

* Adds the elastic and kibana passwords to the .env file for cluster tests

* Get the elastic password later in the pipeline

* Run cluster on every run, don't clean up Azure resources

* Change the Kibana URL to 5601

* Pass the Elastic password to the check_agent_reporting.sh script

* Adding beginning docs for cloud

* Moving dev notes to their own docs page

* Deleting old docs from the current iteration

* Delete old docs/chapters from docs/ directory

* Adding updates to docs

* Fix Readme links and references

* Export the Elastic password to the check_agent_reporting.sh script

* Adds ability to add an external windows server to the network

* Update the workflows to run at the right times

* Remask the secrets in the pipeline

---------

Co-authored-by: Michael Reeves <[email protected]>

* add volume management doc

* Update volume-management.md

* Update volume-management.md

* Fix download zip command

* Update README.md

* add index lifecycle docs

* Update index-management.md

* Update index-management.md

* Update index-management.md

* Update index-management.md

* Update index-management.md

* delete image

* Remove some old TODOs

* Don't remove the Azure resources at the end of the Linux only tests

* Change the variables to work with the 2.0 pipeline

* Run the cluster run workflow to debug the tests

* Run ansible playbook to set fleet

* Log secrets in set_fleet.yml

* Debug setting the environment variables in set_fleet.yml

* Set the debug mode in set_fleet.yml

* Log the Fleet API call details in set_fleet.yml

* Loop through Fleet API calls in set_fleet.yml

* Change the Fleet API call to loop through attempts in set_fleet.yml

* Try to set the Fleet API with retries in set_fleet.yml

* Attempt to output the Fleet API call details in set_fleet.yml

* A new way to handle the Fleet API call in set_fleet.yml

* Export the check_fleet_api.yml file in set_fleet.yml

* Make sure the password is in the check_fleet_api.yml file

* Exit the loop if the Fleet API call succeeds in check_fleet_api.yml

* Wait a little longer for the results to be written to the index

* Update the cluster.yml workflow to wait a little longer for the results to be written to the index

* Delay in a different way

* Attempt a different looping method

* Remove the set-fleet script from the installer

* Reverts to old loop method

* Check that fleet is ready in an external script

* Call the ansible playbook from the install script

* Get the CA fingerprint from the Elasticsearch container

* Adds headers to the curl commands in the set_fleet.yml playbook

* Address the hosts and fleet API issues

* Change the way we login to the Kibana API

* Increase the timeout for the Endpoint Policy API calls

* Increase the timeout for the Endpoint Policy API calls

* Increase the timeout for the Defend Policy API calls

* Only print debug information if debug_mode is true

* Keeps the azure resources on builds

* Fixing Error with certs where the permissions should only be on first
generation!

* Remove sysctl edits to lower privileged ports and add 443 to kibana container

* Add notes on starting vms via azure cli to testing v2

* Fix ansible errors in checking for passwords that are created

* Add debugging commands, and remove references to 443 for kibana from
debug commands

* Update the cluster.yml file to use the new IP address for the Azure instance

* Only allow the ip address of the host to connect to the azure instance

* remove unnecassary script

* Move ansible files to the ansible directory

* Moving Upgrade Readme into upgrade directory

* Add upgrading docs and remove dev notes

* Update main readme docs:

- add section for LME introductory content
- disclaimer around small simple siem
- add Pre-Requisites page
- add Upgrading 1.4 -> 2.0 docs
- note on lme-frontend coming later
- remove references to lmed and make docs accurate to current
  installation process
- add TODOs for docs that still need updated

* Adding updated cloud docs and firewall explanation

* Update FAQ and Pre-requisites

---------

Co-authored-by: Cbaxley and Michael Reeves <[email protected]>

* Remove memory limitations in kibana/elasticsearch quadlets

* adding starting dashboards docs + scripting

* adding starting dashboards docs + scripting

* Fixing passwords in init-setup

* Uploading first setup of post-install.yml

* Removing old dashboard

* Uplloading updated documentation

* Remove optional passowrd setting and add in small changes

* Updating post_install to include wazuh reset and readonly_user creation

* Adding notes on manual passwords:
- leaving notes on this, but stating its unsupported

* pushing ansible changes to expand path and run script to change wazuh password

* Automatic sysmon install powershell script

* Documentation on installing sysmon

* add wazuh documentation

* Update wazuh-configuration.md

* Update wazuh-configuration.md

* add agent management documentation

* update wazuh

* wazuh active reponse doc

* Update elastic-agent-mangement.md

* sysmon install and auditd install

* update

* Remove memory limitations in kibana/elasticsearch quadlets

* adding starting dashboards docs + scripting

* adding starting dashboards docs + scripting

* Fixing passwords in init-setup

* Uploading first setup of post-install.yml

* Removing old dashboard

* Uplloading updated documentation

* Remove optional passowrd setting and add in small changes

* Updating post_install to include wazuh reset and readonly_user creation

* Adding notes on manual passwords:
- leaving notes on this, but stating its unsupported

* starting to integrate notes

* Update install-auditd.md to include script

* update faq and troubleshooting

* formatting agent section

* Push documentation changes:

- remove old irrelevant TOC links to rest of docs
- setup agent/tool docs
- fix up missing pieces
- add docs for certificates.md,upgrading,faq,troubleshooting

* Adding elastalert2

* Adding elastalert container

* Adding elastalert2 config

* add health checks for fleet server

* add health checks to wazuh

* Updating diagram, and readme wtih some clarifying changes

* Move faq back to its original spot

* refactored dashboards, adding divines dashboards

* Adding in powershell script from grant + updated Readme

* ADd back fleet with healthcheck

* Add Wazuh with health check

* Add support for backups

* Remove todo under backups

* adjust dashboard names

---------

Co-authored-by: awarz <[email protected]>
Co-authored-by: Andrew Arz <[email protected]>
Co-authored-by: Connor Aubry <[email protected]>
Co-authored-by: Diabe <[email protected]>

merging post install into the pipeline

* folder for refactored dashboards to be filled in

* Added clause on MANAGER_IP in README

* Command to check linux wazuh agent error

* Refactored User Security

* refactored dashboards

* changes to dashboard panel

---------

Co-authored-by: Brown <[email protected]>
Co-authored-by: Diabe <[email protected]>
Co-authored-by: Clint Baxley <[email protected]>

* Updates some obscure bugs on install

* Adds waiting before retrying to check if the agent is reporting

* Increment the counter before retrying to check if the agent is reporting

* Fail if the elastic agent enrollment fails

Deleting the dashboard refractor directory from the release-2.0.0 branch...

# Conflicts:
#	.devcontainer/python_development/devcontainer.json
#	.github/ISSUE_TEMPLATE/bug-or-error-report.md
#	.github/PULL_REQUEST_TEMPLATE.md
#	.github/workflows/cluster.yml
#	.github/workflows/linux_only.yml
#	.gitignore
#	Chapter 1 Files/Group Policy Objects/{3B4DDB7C-64A7-4E0D-AC21-9FEC36F944B0}/DomainSysvol/GPO/Machine/Preferences/Groups/Groups.xml
#	Chapter 1 Files/Group Policy Objects/{3B4DDB7C-64A7-4E0D-AC21-9FEC36F944B0}/DomainSysvol/GPO/Machine/Preferences/Services/Services.xml
#	Chapter 1 Files/Group Policy Objects/{3B4DDB7C-64A7-4E0D-AC21-9FEC36F944B0}/DomainSysvol/GPO/Machine/comment.cmtx
#	Chapter 1 Files/Group Policy Objects/{3B4DDB7C-64A7-4E0D-AC21-9FEC36F944B0}/DomainSysvol/GPO/Machine/microsoft/windows nt/SecEdit/GptTmpl.inf
#	Chapter 1 Files/Group Policy Objects/{ADC94B57-F661-4DF6-9876-69CC352B6D27}/DomainSysvol/GPO/Machine/Preferences/Services/Services.xml
#	Chapter 1 Files/Group Policy Objects/{ADC94B57-F661-4DF6-9876-69CC352B6D27}/DomainSysvol/GPO/Machine/comment.cmtx
#	Chapter 1 Files/Group Policy Objects/{ADC94B57-F661-4DF6-9876-69CC352B6D27}/DomainSysvol/GPO/Machine/microsoft/windows nt/SecEdit/GptTmpl.inf
#	Chapter 1 Files/Group Policy Objects/{ADC94B57-F661-4DF6-9876-69CC352B6D27}/DomainSysvol/GPO/Machine/registry.pol
#	Chapter 1 Files/lme_gpo_for_windows.zip
#	Chapter 1 Files/lme_wec_config.xml
#	Chapter 3 Files/dashboard_update.sh
#	Chapter 3 Files/deploy.sh
#	Chapter 4 Files/dashboards/alerting_dashboard.ndjson
#	Chapter 4 Files/dashboards/computer_software_overview.ndjson
#	Chapter 4 Files/dashboards/healthcheck_dashboard_overview.ndjson
#	Chapter 4 Files/dashboards/process_explorer.ndjson
#	Chapter 4 Files/dashboards/security_dashboard_security_log.ndjson
#	Chapter 4 Files/dashboards/user_hr.ndjson
#	Chapter 4 Files/dashboards/user_security.ndjson
#	README.md
#	dashboards/elastic/sysmon_summary.ndjson
#	docs/markdown/chapter1/chapter1.md
#	docs/markdown/chapter1/guide_to_ous.md
#	docs/markdown/chapter2.md
#	docs/markdown/chapter3/chapter3.md
#	docs/markdown/chapter3/resilience.md
#	docs/markdown/chapter4.md
#	docs/markdown/logging-guidance/other-logging.md
#	docs/markdown/maintenance/backups.md
#	docs/markdown/maintenance/certificates.md
#	docs/markdown/maintenance/upgrading.md
#	docs/markdown/prerequisites.md
#	docs/markdown/reference/dashboard-descriptions.md
#	docs/markdown/reference/troubleshooting.md
#	testing/InstallTestbed.ps1
#	testing/configure/azure_scripts/download_in_container.ps1
#	testing/configure/linux_install_lme.exp
#	testing/development/docker-compose.yml
#	testing/development/upgrade_lme.sh
#	testing/merging_version.sh
#	testing/tests/.env_example
#	testing/tests/README.md
#	testing/tests/api_tests/data_insertion_tests/conftest.py
#	testing/tests/api_tests/data_insertion_tests/test_server.py
#	testing/tests/api_tests/helpers.py
#	testing/tests/api_tests/linux_only/conftest.py
#	testing/tests/api_tests/linux_only/test_server.py
#	testing/tests/api_tests/winlogbeat/conftest.py
#	testing/tests/api_tests/winlogbeat/test_server.py
#	testing/tests/selenium_tests/cluster/conftest.py
#	testing/tests/selenium_tests/cluster/test_computer_software_overview_dashboard.py
#	testing/tests/selenium_tests/cluster/test_health_check_dashboard.py
#	testing/tests/selenium_tests/cluster/test_process_explorer_dashboard.py
#	testing/tests/selenium_tests/cluster/test_security_dashboard_security_log.py
#	testing/tests/selenium_tests/cluster/test_sysmon_summary_dashboard.py
#	testing/tests/selenium_tests/cluster/test_user_h_r_dashboard.py
#	testing/tests/selenium_tests/cluster/test_user_security_dashboard.py
#	testing/tests/selenium_tests/linux_only/conftest.py
#	testing/tests/selenium_tests/linux_only/test_basic_loading.py
#	testing/tests/selenium_tests/linux_only/test_computer_software_overview_dashboard_lo.py
#	testing/tests/selenium_tests/linux_only/test_health_check_dashboard_lo.py
#	testing/tests/selenium_tests/linux_only/test_security_dashboard_security_log_lo.py
#	testing/tests/selenium_tests/linux_only/test_sysmon_summary_dashboard_lo.py
#	testing/tests/selenium_tests/linux_only/test_user_h_r_dashboard_lo.py
#	testing/tests/selenium_tests/linux_only/test_user_security_dashboard_lo.py

* Fixing and integrating changes from dashboard_fix branch:

  - remove dashboard_refactor folder
  - add in wazuh dashboards
  - add in quadlets

* Adding in andrews dashboard changes

* update wazuh vulnerability dashboard

* Rename wazuh_amazon_aws_alerts.dumped.ndjson to wazuh_amazon_aws_alerts.ndjson

* Rename wazuh_docker_listener_events.dumped.ndjson to wazuh_docker_listener_events.ndjson

* rename dashboards

* put dumped back in

* delete old dashboard file

* fixed dashboard panels and removed links from dashboard menu panel

* updating comment to reflect accurate 30 minute timeout

* Adds azure region selector to the cluster workflow

* Updates the container images to the latest versions

* Updates the container images to the latest versions in all relevant files

* Updates the test_server.py to match the latest version

* Debug why the server won't start with the new versions

* Follow redirects when checking if the services are up

* Removing debugging for the linux_only workflow

* Update all the curl calls to use -L

* remove links, remove extra security dashboard, fix name of dashboard

* Add links and tags for elastic dashboards

* fix location of dashboards

* delete extra dashboard

* remove dupe id's

* revert dashboards to before dupe id removal script

* ALL fixes for LME dashboards

* fix again...

* update conf mount

* Updaing wazuh configuration to support vuln scanning

* Update lme-wazuh-manager.container

* Push updated Wazuh dashboards AND dashboards docs

* Adding email alert rule example

* Update wazuh dashboards

* Removed 1 symbol in deploy wazuh windows command so it will work on copy-paste

* Update the tests for the new LME version to skip the indices that were changed

* Set a default region for the cluster workflow

* Remove cluster run from automatic running.

* Adds a sleep before running the tests

---------

Co-authored-by: Michael Reeves <[email protected]>
Co-authored-by: ddiabe <[email protected]>
Co-authored-by: cbaxley <[email protected]>
Co-authored-by: Grant (SNL) <[email protected]>

* seltest rel2 final2

* added test case for policy changes dashboard

* typo changes on policy changes dashboard

* updates on failed api & sel tests

* Update cluster.yml

* test changes to pass cicd

* Update cluster.yml

* adding in lme-2-docs content

* add includes.txt for building AND all the new pngs

* adding the new includes.txt and removing space named file

* adding main readme from lme-2-docs, add .gitignore