diff --git a/docs/devops-guidelines/.images/DevOps.png b/docs/devops-guidelines/.images/DevOps.png new file mode 100644 index 00000000..5bcce4c3 Binary files /dev/null and b/docs/devops-guidelines/.images/DevOps.png differ diff --git a/docs/devops-guidelines/.order b/docs/devops-guidelines/.order new file mode 100644 index 00000000..ba05b3ee --- /dev/null +++ b/docs/devops-guidelines/.order @@ -0,0 +1,8 @@ +DevOps-Guidelines +Plan +Create +Verify +Deploy +Configuration-Management +Monitoring +Package \ No newline at end of file diff --git a/docs/devops-guidelines/Configuration-Management.md b/docs/devops-guidelines/Configuration-Management.md new file mode 100644 index 00000000..806d8edb --- /dev/null +++ b/docs/devops-guidelines/Configuration-Management.md @@ -0,0 +1,36 @@ +[DevOps Strategy](/Project-Artifacts/DevOps-Strategy) + +**Capabilities Inventory** + +|Capability|Ideal State|Tools In Inventory| +|--|--|--| +|Infrastructure Provisioning|Infrastructure is provisioned and configured as code along with the rest of the workload in an automated CI/CD pipeline|Terraform, AWS CloudFormation, Azure Resource Manager| +|Infrastructure Configuration|Infrastructure is provisioned and configured as code along with the rest of the workload in an automated CI/CD pipeline|Terraform, AWS Config, AWS CLI, Azure Policy, Azure CLI| +|Application Provisioning|Provision via Continuous Deployment pipeline using GitHub Actions, Azure Pipelines, or similar; users and administrators provisioned with proper access|Azure DevOps, GitHub| +|Application Configuration|Configuration settings are exposed based on user roles. Changes to configuration items are audited and sent through and appropriate approval process.|| + +**Infrastructure Provisioning** +- Infrastructure will be provisioned using Terraform templates. This will allow the team to standardize on a tool several team members already know that the company is licensed for, is supported for common cloud platforms and provides provisiong of most native capabilities. + +- All provisioning templates regardless of technology will be stored in git repos and deployed use the pipeline toofl of choice such as GitHub Actions or Azure DevOps Pipelines. + +- Infrastructure provisioning should follow [resource naming guidelines](todo ) and [tagging guidelines](todo ) . + +**Infrastructure Configuration** +- Since all resources will be deployed and configured via code (Terraform, CloudFormation, etc.), configurations will be stored in version control (Git Repo). + +- Any updates will be reviewed via a Pull Request, then can be deployed by the service of choice. Terraform has the capability to update existing resources or replace resources that cannot be updated in place. + +It is critical that any resource that maintains state (databases, blob storage, aka a stateful resource) be backed up and restorable in the event of a resource replacement, or, stateful resources should be excluded from full automated provisioning. + +**Application Provisioning** +- Applications should be provisioned via automated pipelines +- Pipeline templates will be stored in git repos along with the the application code + +**Application Configuration** +- Non secret application configuration information should be stored with source code. + +- Sensitive information, passwords or API keys, should be stored in a cloud native secret store, or, the pipeline tool of choice. + +- Secrets based only on deployment environment can be + diff --git a/docs/devops-guidelines/Create.md b/docs/devops-guidelines/Create.md new file mode 100644 index 00000000..c3a50e51 --- /dev/null +++ b/docs/devops-guidelines/Create.md @@ -0,0 +1,63 @@ +[[_TOC_]] + +Specific tools used to create the source and related artifacts will be determined by a project team. + +These guidelines are intended to be general guidance and will be applicable to most teams. + +# Managing Artifacts +For technical and software projects, a variety of artifacts are created. All artifacts should be maintined in git based repos + +- Design artifacts such as high level designs, information architecutre, wire frames and similar should be committed to source and associated with a User Story whenever possible +- Software developers should create code in a branch created from a User Story or Bug/Defect + +todo: Branching Strategies +todo: Repository Structure + +**Capabilities Inventory** + +|Capability|Ideal State|Typical Tools| +|--|--|--| +|Design|Designs are driven by user stories, captured ahead of development and vetted against a Definition of Ready (DoR) for design acceptance. Design documents incorporate all aspects of the design, to include at a minimum UX/UI, infrastructure, data models, logic and integration. Design may optionally include a clickable prototype or a proof of concept.|Adobe XD, Zeplin, Adobe Photoshop, Adobe Illustrator| +|Implementation|Developers work on tasks tied to user stories. Unit tests are developed for acceptance criteria defined in the design ahead of development of the actual product. Tasks are completed in priority order. Development is accelerated by the use of IDEs and feedback tools.|Visual Studio, Visual Studio Code, MuleSoft, SQL Server Management Studio, Oracle SQL Developer, PL/SQL Developer, Resharper +|Version Control|All artifacts are versioned and revisions to those artifacts are tracked by a version control system (git). Merges are simple and reversion to a "known-good state" is easy. The version control system supports distributed development and forms the backbone for the build/deployment pipelines.|GitHub, git based Azure DevOps Repos, et.al. | + +# Design +Designs typically consist of wireframes, mockups or prototypes. +- A wireframe is a low-fidelity visual representation of the site and will be used to depict the basic UI, detailing the layout and structure of the site. +- A mockup is a mid to high-fidelity visual representation of the site and will be used to highlight color scheme, visual style and typography. +- A prototype is a high-fidelity visual representation of the site and will be used to simulate user interaction by allowing the user to experience the content through clickable elements and interactions. + +## Design Guidelines +Use the design that meets your needs. We recommend starting with wireframes, using mockups and prototypes when necessary and their value is clear. + +- Designs should utilize an existing design system such as Material Design, as well as standardized componenents such as Font Awesome Icons and custom components. +- Do not develop eponymous designs unless they are truly required for differentiation. +- Beware existing or complementary solutions. +- Regardless of design, usability should not be sacrified: value something that works well over somthing that looks pretty +- When possible, include user testing in design review. + +Designs should be reviewed by stakeholders and users, as well as the design team, development team and business analysts before finalization. User stories, acceptance criteria and UX personas are used to approve and validate designs. + +# Implementation +Implementation is where the rubber meets the road. The plans are laid. The user stories are written. The test cases have been identified and it's finally time to build something. + +Implementation will follow the SCRUM methodology and utilize Test Driven Development (TDD). All features will be logged as user stories with acceptance criteria before they are worked. + +All development is expected to be performed using the appropriate development environmetn such as Visual Studio, Visual Studio Code, Eclipse, ... + +## Source Management +Git repos in Azure DevOps are preferred. + +Associate commits using user stories using the syntax "# ..." in commit messsages to track which user story the commit is for. This generates a source audit trail and can be used by release systems to track which stories are being delivered in a release. + +Code should be committed on a regular basis. More frequent commits provide benefits: +- Team members can merge committed changes more often reducing overall merge effort +- Continous integration builds can be triggered to verify code when it is committed + +## Coding Standards +Developers will follow the appropriate coding standards for the languages being used. Coding standards can be found at + +## Unit Tests +Unit tests will be developed for each user story that comply with the and are adequate to test the feature. + + diff --git a/docs/devops-guidelines/Create/Branching-Strategy.md b/docs/devops-guidelines/Create/Branching-Strategy.md new file mode 100644 index 00000000..e69de29b diff --git a/docs/devops-guidelines/Create/Repository-Structure.md b/docs/devops-guidelines/Create/Repository-Structure.md new file mode 100644 index 00000000..0707cafc --- /dev/null +++ b/docs/devops-guidelines/Create/Repository-Structure.md @@ -0,0 +1,104 @@ +--- + +# Introduction + +One of the bigger challenges to building software is managing the lifecycle of the source code, from authorship to deployment. Below is an outline of how to structure our source code, the goal of which is to achieve the following: + +- Any change must be easily reviewable, including how the change affects dependencies. +- Building and deploying should be simple, fast, and reproducible. +- Changes should be testable in both isolation and integrated with the existing environment. +- Setting up new users/systems should require as few steps as possible and require little outside assistance. + +--- + +# Review + +There are generally two approaches when setting up source control for a project: + +- Each application's code remains in its own repository +- All code is in one repository (monorepo) + +Neither implementation is objectively better than the other; they are both highly dependent on the complexity of the application and delivery model. Below is a review of the benefits and challenges that come with each approach. + + +## Separate Repositories + +### Benefits + +- Organization +The structure of the repo is simple; it contains the source files and configuration from only one project. +- Isolation +Each project exists in isolation. Changes can be made, and subsequently reviewed, knowing they have no direct or immediate impact on anything else. + +### Challenges + +- Dependencies +Coordinating changes between multiple, dependent projects is a tedious process. It is made worse over a project's lifecycle as it naturally grows in complexity. One example of this would be a web app that is dependent on a component library. When the app requires a change to a component, the dependent component repo must go through its code review and release process before the web app is able to move forward. If there are further changes required of the updated component library, the cycle starts over and the delay increases. +- Deployments +The ability to automate build and release pipelines suffers when the dependency graph grows in complexity. Coordinating deployments requires a significant effort from the team to track interdependent versions and where they should exist in the release cycle. + + +## Single Repository (monorepo) + +### Benefits + +- Organization +All files exist in one, well-structured directory. Users can easily search across projects without having to sync updates and branches across multiple repositories. +- Dependencies +Every project moves forward together. Changes in dependencies should never block progress since review cycles and upstream integration never happen orthogonally. +- Tooling +Most applications and services operate on the file system. Providing immediate access to all required files greatly improves the speed, efficiency, and resiliency of the CI/CD pipelines. + +### Challenges + +- Scalability +The life of a repository is long-lived. The VCS index continues to grow with every new file or changed line. While there are ways to mitigate this, the size of the repo will inevitably grow. + +--- + +# Model Repository Implementation + +Our project is comprised of multiple slices of related functionality and will be developed in parallel by a distributed team. The current approach will maintain a single repository ("monorepo") containing top-level directories for each independent project. + +The structure of the repository will be similar to the following example: + +- **Root** (repo directory) + - Repo-level config files: e.g. `.gitignore`, `.gitattributes`, `.editorconfig` + - Makefile (or similar) to provide top-level hooks into available scripts + - **docs** + - README.md + - installation.md + - **scripts** (utilities for cross-project coordination) + - README.md + - Makefile + - pipeline + - build.sh + - test.sh + - utils + - setup_machine.sh + - **infrastructure** + - README.md + - Makefile + - env + - ring1 + - ring2 + - modules + - lambda + - s3 + - **frontend** + - Project-level config files: e.g. `.gitignore`, `.npmrc`, `setup.cfg` + - docs + - README.md + - installation.md + - scripts + - build.sh + - test.sh + - src + - tests + - **api** + - **shared_components** + - **bloomreach** + + +--- + diff --git a/docs/devops-guidelines/Deploy.md b/docs/devops-guidelines/Deploy.md new file mode 100644 index 00000000..6a0930b7 --- /dev/null +++ b/docs/devops-guidelines/Deploy.md @@ -0,0 +1,24 @@ +[[_TOC_]] + +Project code and artifacts are built locally using tools dictated by the software architecture of a project. + +Regardless of architecture, projects should automated builds to support *Continuous Integration* or *CI*, and, automated deployment to staged environments + +# Continuous Integration +Build the solution on every commit. Build should inluding executing and passing all unit tests as well as security code scans. + +# Continous Deployment +When software is succesfully built, it should be deployed immediately. It is most common to deploy in stages so that deploymnets can be verified and tested. + +|Stage | Description | +|-------|----------| +|Dev |Deploy to dev on every successful build of the system. Run integration tests.| +|Test |Proceeds whenever Dev passes all tests. Additional integration tests are performed. In some cases, manual tests or automated UI tests are executed, or, 3rd party test systems are included in integration tests| +|Pre production|As close to production as reasonable. May have a copy of production data to test migrations on and/or run additional production tests| +|Production|final production environment or environments | + +## Ring Deployment Model + +Often, deployment to Production isn't deployment to a single instance of a system. when multiple systems exist, supporting multiple regions around the world for example, a ringed deployment should be used. This allows for controlled rollouts and limits the impact of deployment issues only discovered as + +Supports rolling and red/green deployments. \ No newline at end of file diff --git a/docs/devops-guidelines/DevOps-Guidelines.md b/docs/devops-guidelines/DevOps-Guidelines.md new file mode 100644 index 00000000..c0aa119e --- /dev/null +++ b/docs/devops-guidelines/DevOps-Guidelines.md @@ -0,0 +1,20 @@ +# DevOps Approach + +"DevOps is cultural fusion of People, Process, and Technology to accelerate delivery of Business Value" + +![DevOps Continous Improvement](.images/DevOps.png) + +A complete DevOps approach combines critial develop, operations and management concerns so that software projects can continously and consistently deliver high quality solutions. + +Each practice within these guidelines works with others to form a working model for continous improvement. In addition, DevOps works hand in hand with portfolio and project management and agile initiatives. + +This wiki is intended for the practitioners: team members on projects. It provides a set of guidelines that can be used as a starting point for any project's DevOps implementation. Each project will need to define specific tools and processes that will work best for that project. This wiki can be cloned and used as a starting point for a project specific DevOps Strategy. + +Topics covered in this site include: +- [Planning and managing work from inception to delivery](/Plan) +- [Source management](/Create) +- [Verifying quality](/Verify) +- [Build and release automation](/Deploy) +- [Monitoring a solution in production](/Monitor) + +This site assumes you are using and are familiar with [GitHub](https://github.com) or [Azure DevOps](https://dev.azure.com). diff --git a/docs/devops-guidelines/Monitor.md b/docs/devops-guidelines/Monitor.md new file mode 100644 index 00000000..f0ecfa20 --- /dev/null +++ b/docs/devops-guidelines/Monitor.md @@ -0,0 +1,26 @@ +[DevOps Strategy](/Project-Artifacts/DevOps-Strategy) + +**Capabilities Inventory** + +|Capability|Ideal State|Tools In Inventory| +|--|--|--| +|Performance Monitoring|Well established baselines and KPIs that can be tested against, monitored, and improved as a part of CI/CD pipeline and release cycles|AWS CloudWatch, Microsoft SCOM| +|Security Monitoring|Automatically monitor and alert for appropriate credential rotation events, critical vulnerability patches, and user activity anomalies|AWS CloudTrail, SkyHigh, Dome9, QRadar| +|Feedback Management|Build feedback submission tools into user interfaces to obtain feedback directly from customers; leverage Product Owner to translate feedback into feature-oriented user stories to be captured and prioritized by planning tools for future releases|Azure DevOps| +|Consumption Management|Utilize monitoring tools to visualize and report consumption metrics including costs. Use standard resource tags to define and aggregate costs|Azure Monitor, AWS CloudHealth, Splunk, ...| +|Telemetry|Data should be captured for all of the major types of telemetry data, business metrics, application metrics, infrastructure metrics and deployment pipeline metrics. Telemetry data is analyzed and leveraged as KPIs to improve the product as part of the feedback loop.| | + +**Performance Monitoring** +todo + +**Security Monitoring** +todo + +**Feedback Management** +todo + +**Consumption Management** +todo + +**Telemetry** +todo diff --git a/docs/devops-guidelines/Package.md b/docs/devops-guidelines/Package.md new file mode 100644 index 00000000..0dc06b2b --- /dev/null +++ b/docs/devops-guidelines/Package.md @@ -0,0 +1,29 @@ +**Capabilities Inventory** + +|Capability|Ideal State|Tools In Inventory| +|--|--|--| +|Package Configuration|Packages are managed by a package manager which groups software together into a release. This will include infrastructure, application code and external libraries. All artifacts are tracked by version. Dependencies between various components and versions are tracked to reduce the risk of errors and create a repeatable build process.|Azure DevOps| +|Approval Workflow|Releases are managed by approvals at each quality gate.  Nothing moves without an approval.  Approval workflow supports delegation and escalation.|Azure DevOps, ServiceNow| + + +**Package Configuration** +Packages (builds) and deployments will be automated with Azure DevOps Pipelines and assigned an automated version that increments with each build. Azure DevOps creates reliable builds on all platforms through integration with Azure Repos using Git for pull requests, checks, and statuses. Code will be built, tested and deployed via Azure DevOps Repos and Pipelines. + +**Approval Workflow** + +The approval workflow provided by Azure DevOps will be used for all flows except the final push to production. + +*Deployments to production will require the creation of a change request in ServiceNow.* + +The workflow will be as follows: + +* Developers can build to their local environment and Ring 1 as needed. +* Developer initiates pull request. +* Development Lead and Architect review code and infrastructure. +* Development Lead and Architect approve code to be merged. +* Merge triggers a build to Ring 1. +* All unit tests and automated scans are triggered. +* If all unit tests and automated scans pass, the Architect and Quality Assurance Lead are notified. +* Quality Assurance Lead and Architect approve deployments to Ring 2 (Integration/Build Testing). +* Quality Assurance Lead approves deployments to Ring 3 (Functional Testing) and Ring 4 (UAT). +* Quality Assurance Lead and Product Owner approve deployment to Ring 5 (Production). diff --git a/docs/devops-guidelines/Plan.md b/docs/devops-guidelines/Plan.md new file mode 100644 index 00000000..7e08e084 --- /dev/null +++ b/docs/devops-guidelines/Plan.md @@ -0,0 +1,36 @@ +[[_TOC_]] + +# Planning and Managing Work + +Value to be delivered by the project should be managed on a backlog + +Project code should be connected to work: Product Backlog Items/User Stories/Issues and Bugs/Defects. + +This enables Azure DevOps/GitHub to automatically generate release and audit reports of work delivered by a deployment. + +## Work Management Tooling + +|Capability|Ideal State|Primary Tool| +|--|-|--| +|Requirements Management|Requirements are captured as Epics/Features/User Stories on a Product Backlog.|Azure DevOps| +|Defect Tracking|Defects are tracked, triaged, prioritized with Product Backlog items.
For addtional information see [See Testing Strategy - Defect Tracking](/Testing-Strategy/Defect-Tracking)|Azure DevOps| +|Incident Management|Incidents and defects should be tracked together. Outages must have a complete root cause analysis with documented actions that will be taken to ensure the risk of a similar outage is reduced in the future.|ServiceNow| +|Release Planning|Release plans are driven by Epics/Features/User Stories in a backlog. Features that are not complete, may need to be disabled (Feature Flag) or held in a Development branch until they are ready. See [Version Control](/Version-Control) for additional information.|Azure DevOps| + +Notes: +- Production defects are created as incidents reported to ServiceNow. If the incident is triaged and determined to be a defect in code, a Bug is created in the appropriate Azure DevOps project. + + +## Release Planning Guidance + +Multiple options exist for managing releases. With any approach, deliverable work (features, partial features, bugs) are defined in a backlog and are delivered in iterations. + +- The simplest approach to release planning is by grouping iterations into a release in Azure DevOps. See [Areas and Iterations](https://docs.microsoft.com/en-us/azure/devops/organizations/settings/about-areas-iterations?toc=%2fazure%2fdevops%2fboards%2fsprints%2ftoc.json&%3bbc=%2fazure%2fdevops%2fboards%2fsprints%2fbreadcrumb%2ftoc.json&view=azure-devops). +- Larger teams with dependencies can leveral plans as described in [Plans (Agile at scale)](https://docs.microsoft.com/en-us/azure/devops/boards/plans/?toc=%2Fazure%2Fdevops%2Fboards%2Ftoc.json&bc=%2Fazure%2Fdevops%2Fboards%2Fbreadcrumb%2Ftoc.json&view=azure-devops) + +A major goal of DevOps and DevOps Strategy is to progress to Continuous Deployment (CD). True continous deployment significantly reduces the need to plan for major releases. Release planning becomes a part of every feature instead of a special event requiring additional effort. + +Continous Deployment is a goal for most teams. Before getting to CD, release planning is required with releases categorized as major or minor. + +- _Major releases_ typically include new features and capabilities, workflow changes, major changes to interfaces and integration points. A major release typically includes the work of several sprints. Use 3 to 6 sprints as a guideline. +- _Minor releases_ are focused on smaller and incremental changes and fixes. A minor release typically doesn't include signifcant new capabilities. A minor release or system changes more frequently (2 sprints or less). diff --git a/docs/devops-guidelines/README.md b/docs/devops-guidelines/README.md new file mode 100644 index 00000000..635aa83c --- /dev/null +++ b/docs/devops-guidelines/README.md @@ -0,0 +1,9 @@ +# DevOps Project Guidelines +This project includes documentation on DevOps practices and standards to be used by projects + +# Contributing +Clone the repo and update as needed + +VSCode is a recommneded editor but any editor with Markdown will do. + +For simple edits, push directly to master. If you have content you would like reviewed, create a branch and issue a PR. diff --git a/docs/devops-guidelines/Verify.md b/docs/devops-guidelines/Verify.md new file mode 100644 index 00000000..cbca00de --- /dev/null +++ b/docs/devops-guidelines/Verify.md @@ -0,0 +1,70 @@ +[[_TOC_]] + +Verifying quality involves many tools and many steps from proper work management through monitoring issues that may occur in a production environment. + +Standard scope of quality verification in DevOps includes: +1. Automated unit tests of code +2. Performing regular peer based code reviews +3. Using branch policies requiring Pull Requests into the branch be reviewed and pass quality checks +4. Analysing code to check for code quality and security issues +5. Automated integration testing +6. Functional testing +7. Load and performance testing +8. Security testing + +# Verification Strategy +Each project should develop test and verify strategy that is appropriate for the project. Factors include number of users, type of project (public facing web site vs. extranet portal vs. data analysis), data privacy and or risk to the business for data exposure. + +A project cannot simply follow a recipe blindly. + +# Testing Pyramid +Of the techniques mentioned, the most powerful is automated unit testing. If appropriate for your project, it is recommended that this become + +## Unit Test References + + +# Automated Unit Testing + +**Capabilities Inventory** + +|Capability|Ideal State|Tools In Inventory| +|--|--|--| +|Code Review|Code reviews are done consistently using pre-defined standards and objective measures of code and design quality. Code is compared to the agreed to architecture to ensure compliance. Reviews are documented, any required changes are documented and worked. Follow-ups are scheduled and tracked.|Azure DevOps| +|Infrastructure Review|Infrastructure is scanned automatically using pre-defined guardrails. Alerts are generated if a change is outside predefined limits.|AWS CloudWatch| +|Licensing Compliance|All code is scanned automatically on check-in for license compliance. Approved licenses are agreed to and documented. License compliance is also added to the code review process.|| +|Static Quality Analysis|All code is scanned automatically on check-in for quality issues. Quality issues are required to be remedied prior to merging to an upstream branch.|SonarQube| +|Unit and Regression Testing|Complex code and foundational code is covered by unit tests. Complex code and foundational code is automatically identified to ensure compliance. Unit tests are executed as part of the build process. Failing unit tests are required to be fixed prior to merges or releases. Regression tests are composed of the entire suite of unit tests + additional higher-level tests as needed. Regression tests are automatically verified as part of the build process prior to release.|Junit, NUnit, Selenium, WorkSoft| +|Performance Testing|Performance testing is part of the deployment pipeline. All applications are held to documented performance standards.|LoadRunner| +|Security Analysis|Code is automatically scanned to protect against malicious components, detect plaintext credentials, validating appropriate IAM, and monitoring for behavioral anomalies.|| + +**Infrastructure Review** + +- Infrastructure will be reviewed using the code review process as all infrastructure will be created via code. + +- These reviews will be focused on ensuring the services are being configured properly and are in alignment with the architectural guidelines. + +**Licensing Compliance** + +- Licensing compliance will be enforced by manual reviews of any external components or libraries during the code review process. If the enterprise procures an automated license compliance capability, the project will leverage the enterprise tools. + +**Static Quality Analysis** + +- Static Quality Analysis will be part of the build pipeline. SonarQube is expected to be leveraged for this task. + +- Quality metrics will be used to ensure the project is meeting the non-functional quality and security requirements. + + +**Performance Testing** + + +**Security Analysis (Static)** + +- Include Static Security Analysis (Static) as part of an automated Continuous Integration build pipeline. Code analysis capabilities are built in to GitHub, and, several good external tools exist including SonarQube which will report on vulnerabilities and fail builds if scans do not complete. +- Scans should focus on the OWASP Top 10 and the SANS Top 25. + + +**Security Analysis (Dynamic)** +- Security Analysis (Dynamic) should be executed as part of a Continuous Deployment pipeline. + + + diff --git a/docs/devops-guidelines/Verify/.order b/docs/devops-guidelines/Verify/.order new file mode 100644 index 00000000..8c8c3c3a --- /dev/null +++ b/docs/devops-guidelines/Verify/.order @@ -0,0 +1,7 @@ +Azure-DevOps-Testing +Azure-Pipelines-and-Testing +Unit-and-Regression-Testing +UAT-Testing +Defect-Tracking +Performance-Testing +Incident-Management \ No newline at end of file diff --git a/docs/devops-guidelines/Verify/Azure-DevOps-Testing.md b/docs/devops-guidelines/Verify/Azure-DevOps-Testing.md new file mode 100644 index 00000000..1ea1c452 --- /dev/null +++ b/docs/devops-guidelines/Verify/Azure-DevOps-Testing.md @@ -0,0 +1,17 @@ +# Introduction +Test Plans are the easiest way to define and configure manual test cases in Azure DevOps. They increase the visibility of which pieces of functionality are covered and define how those business requirements get tested. + +# Relevant Documentation + +The following documents provide + +- [Test Plans](https://docs.microsoft.com/en-us/azure/devops/test-plans/index) +- [Create Test Plans and Test Suites](https://docs.microsoft.com/en-us/azure/devops/test/create-a-test-plan) +- [Create Manual Test Cases](https://docs.microsoft.com/en-us/azure/devops/test/create-test-cases) +- [Run Manual Tests](https://docs.microsoft.com/en-us/azure/devops/test/run-manual-tests) +- [Azure DevOps Continuous Build/Deploy/Test with ASP.NET Core](https://www.hanselman.com/blog/AzureDevOpsContinuousBuildDeployTestWithASPNETCore22PreviewInOneHour.aspx) + +# Hands-On Labs + +- [Test Planning and Management with Azure Test Plans](https://www.azuredevopslabs.com/labs/azuredevops/testmanagement/) +- [Exploratory Testing with Azure Test Plans](https://azuredevopslabs.com/labs/azuredevops/exploratorytesting/) diff --git a/docs/devops-guidelines/Verify/Azure-Pipelines-and-Testing.md b/docs/devops-guidelines/Verify/Azure-Pipelines-and-Testing.md new file mode 100644 index 00000000..c5fe30b3 --- /dev/null +++ b/docs/devops-guidelines/Verify/Azure-Pipelines-and-Testing.md @@ -0,0 +1,35 @@ +Azure Pipelines is a cloud service that you can use to automatically build and **TEST** your code project and make it available to other users. + +Below are key guides for developers on testing using Azure Pipeline related to **testing** + +[Review test results](https://docs.microsoft.com/en-us/azure/devops/pipelines/test/review-continuous-test-results-after-build?view=azure-devops) +Test reports provide an effective and consistent way to view the tests results executed using different test frameworks, in order to measure pipeline quality, review traceability, troubleshoot failures and drive failure ownership. + +[Analyze test results](https://docs.microsoft.com/en-us/azure/devops/pipelines/test/test-analytics?view=azure-devops) +Tracking test quality over time and improving test collateral is key to maintaining a healthy DevOps pipeline. Test analytics provides near real-time visibility into your test data for builds and releases. It helps improve the efficiency of your pipeline by identifying repetitive, high impact quality issues. + +[Review code coverage!](https://docs.microsoft.com/en-us/azure/devops/pipelines/test/review-code-coverage-results?view=azure-devops) +Code coverage helps you determine the proportion of your project's code that is actually being tested by tests such as unit tests. To increase your confidence of the code changes, and guard effectively against bugs, your tests should exercise - or cover - a large proportion of your code. + +Reviewing the code coverage result helps to identify code path(s) that are not covered by the tests. This information is important to improve the test collateral over time by reducing the test debt. + + + +Other testing documentation + +- [Trace test requirements](https://docs.microsoft.com/en-us/azure/devops/pipelines/test/requirements-traceability?view=azure-devops) + +- [Use Test Impact Analysis](https://docs.microsoft.com/en-us/azure/devops/pipelines/test/test-impact-analysis?view=azure-devops) + +- [UI testing consideration](https://docs.microsoft.com/en-us/azure/devops/pipelines/test/ui-testing-considerations?view=azure-devops&tabs=mstest) + +Run tests in parallel + +- [For any test runner](https://docs.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-any-test-runner?view=azure-devops) + +- [Using VS Test task](https://docs.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-vstest?view=azure-devops) + + + + + diff --git a/docs/devops-guidelines/Verify/Defect-Tracking.md b/docs/devops-guidelines/Verify/Defect-Tracking.md new file mode 100644 index 00000000..6df026c8 --- /dev/null +++ b/docs/devops-guidelines/Verify/Defect-Tracking.md @@ -0,0 +1,9 @@ +**Defect Tracking and Resolution** + +Defect should be tracked in the same way as User Stories. If appropriate, link the original user story to the defect. + +Defects generated from automated testing tool/platform, when validated, should be recorded the same as any ohter defect. + +Defects are reviewed and planned with other user stories on the backlog . + +If a defect is high priority, process shoudl be followed but it may be expedited: Add the defect to the backlog, create a branch for the defect, write unit tests for the defect, fix the code, push the branch, create PR, when approved. Existing CI/CD pipelines should then deploy the fix to production. diff --git a/docs/devops-guidelines/Verify/Incident-Management.md b/docs/devops-guidelines/Verify/Incident-Management.md new file mode 100644 index 00000000..0e118107 --- /dev/null +++ b/docs/devops-guidelines/Verify/Incident-Management.md @@ -0,0 +1,13 @@ +**Incident Management** + +Incidents are defects found after a feature has rolled out to production. + +Once a feature has been productionized (Ring 5), any issue found will be logged in the ServiceNow incident tracking system. Service delivery groups will be defined in Service Now that will immediately notify the appropriate DCX project/support team of reported issue. + +Service Delivery groups will contain the name(s) of specific personnel assigned to address issues in various areas of the overall DCX solution (UI, CMS, Integration etc.) + +Incidents logged in ServiceNow will default to the current escalation process for all incidents in ServiceNow. + +For Bizlink external customers, the Global Marketing help desk will continue to log cases in Salesforce, however any technical issues will be logged in ServiceNow as is their current practice. + +DCX project will work with the Service Now team on exposing an API to the new DCX solution, where external customers will have the option of submitting issues directly to Phillips 66. The ServiceNow tickets will be directed to either the Global Marketing help desk or the Lubes Customer Excellence team for review. diff --git a/docs/devops-guidelines/Verify/Performance-Testing.md b/docs/devops-guidelines/Verify/Performance-Testing.md new file mode 100644 index 00000000..e69de29b diff --git a/docs/devops-guidelines/Verify/UAT-Testing.md b/docs/devops-guidelines/Verify/UAT-Testing.md new file mode 100644 index 00000000..e69de29b diff --git a/docs/devops-guidelines/Verify/Unit-and-Regression-Testing.md b/docs/devops-guidelines/Verify/Unit-and-Regression-Testing.md new file mode 100644 index 00000000..95cbcaac --- /dev/null +++ b/docs/devops-guidelines/Verify/Unit-and-Regression-Testing.md @@ -0,0 +1,15 @@ +**Unit and Regression Testing** + +Developers will use Test-driven development (TDD) technique which combines test-first development where you write a test before you write just enough code to fulfill that test. + +The benefits of Test-driven development are more than just simple validation of correctness but can also drive the design of the code ensuring it meets the test case. + +Unit tests are executed as part of the build process. Failing unit tests are required to be fixed prior to merges or releases. Test-driven development creates a regression-test suite as a side effect that can minimize human manual testing, while finding problems earlier, leading to quicker fixes. + +Regression tests are composed of the entire suite of unit tests + additional higher-level tests as needed. Regression tests are automatically verified as part of the build process prior to release. + +A good overall testing strategy should rely heavily on unit testing but may be augmented by automated UI testing and manual testing where required. + + + +