Skip to content

US Digital Services Playbook Play 10

Jump to bottom
mattkwong-kpmg edited this page Mar 3, 2017 · 8 revisions

#Play 10: Automate Testing and Deployments Today, developers write automated scripts that can verify thousands of scenarios in minutes and then deploy updated code into production environments multiple times a day. They use automated performance tests which simulate surges in traffic to identify performance bottlenecks. While manual tests and quality assurance are still necessary, automated tests provide consistent and reliable protection against unintentional regressions, and make it possible for developers to confidently release frequent updates to the service.

##Checklist

  • 1. Create automated tests that verify all user-facing functionality

We have begun creating automated end-to-end user interface tests using Protractor (consult the Development Guide for more details).

  • 2. Create unit and integration tests to verify modules and components

As part of our Definition of Done, unit and integration tests must be checked in with code changes. We use ESLint for static analysis, Jasmine as the test framework, Karma for running unit tests, Protractor for end-to-end tests, and Istanbul for capturing code coverage (consult the Development Guide for more details). Due to time constraints, the unit tests do not provide the level of code coverage that we would typically like to have.

  • 3. Run tests automatically as part of the build process

As part of our Jenkins Continuous Integration process, all of the tests described above are triggered automatically once a developer checks code into their feature branch on Github, and the results of the build pipeline are automatically pushed to a slack channel for the team to review.

  • 4. Perform deployments automatically with deployment scripts, continuous delivery services, or similar techniques

As part of our Jenkins Continuous Integration process, once the technical architect approves and merges a code change to the master branch, the newly updated master branch is built and automated tests are re-run. If all tests pass, the new build is automatically deployed to the development environment. A similar process, including approval from the Product Owner, is used to automatically deploy a build to the testing and production environments. This is all accomplished using Github, Jenkins, and Docker.

  • 5. Conduct load and performance tests at regular intervals, including before public launch

Not necessary for this prototype

##Key Questions

  1. What percentage of the code base is covered by automated tests?

Istanbul is used to measure and report this information. These reports are captured as a Jenkins artifacts for each build. For future sprints we plan to improve the current coverage score reported by Istanbul.

  1. How long does it take to build, test, and deploy a typical bug fix?

Not including the Technical Architect approval that must occur, an automated build, test, and deploy to the Development Environment takes about twenty minutes.

  1. How long does it take to build, test, and deploy a new feature into production?

Not including the Product Manager approval that must occur, an automated build, test, and deploy to Production takes about twenty minutes.

  1. How frequently are builds created?

As part of our Continuous Integration process, a build is created every time a developer commits code.

  1. What test tools are used?

We are using a customized front end that utilizes ESLint, Jasmine, Istanbul, Karma, and Protractor.

  1. Which deployment automation or continuous integration tools are used?

We are using Github, Jenkins, and Docker as the core components of our continuous integration process.

  1. What is the estimated maximum number of concurrent users who will want to use the system?

Not considered as part of this prototype.

  1. How many simultaneous users could the system handle, according to the most recent capacity test?

Not evaluated as part of this prototype.

  1. How does the service perform when you exceed the expected target usage volume? Does it degrade gracefully or catastrophically?

The team is monitoring usage volume in real-time using Nagios alerts which would give the team time to take corrective action. For the purposes of the prototype, however, a graceful fail-over mechanism has not been implemented. This would be a target for a future sprint.

  1. What is your scaling strategy when demand increases suddenly?

The Docker container can be deployed to additional machines and registered with the load balancer & monitoring systems

#US Digital Services Playbook

  1. Play 1 Understand what people need
  2. Play 2 Address the whole experience, from start to finish
  3. Play 3 Make it simple and intuitive
  4. Play 4 Build the service using agile and iterative practices
  5. Play 5 Structure budgets and contracts to support delivery
  6. Play 6 Assign one leader and hold that person accountable
  7. Play 7 Bring in experienced teams
  8. Play 8 Choose a modern technology stack
  9. Play 9 Deploy in flexible hosting environment
  10. Play 10 Automate testing and deployments
  11. Play 11 Manage security and privacy through reusable processes
  12. Play 12 Use data to drive decisions
  13. Play 13 Default to open
Clone this wiki locally