Skip to content

Latest commit

 

History

History
145 lines (99 loc) · 5.12 KB

M2.md

File metadata and controls

145 lines (99 loc) · 5.12 KB
Raw

Pipeline > Test+Analysis

🧪 Mutation coverage with snapshot testing 📸

🚰 You will be creating a testing analysis that will blend several techniques, including fuzzing, and snapshot testing and integrating into your pipeline tool.

The goal of the analysis is to calculate the mutation coverage of a test suite.

# Provision and configure computing environment for pipeline
$ pipeline init
<connection information>

# Trigger a build job, running steps outlined by build.yml, wait for output, and print build log.
$ pipeline build mutation-coverage build.yml
<mutation report log>
MUTATION COVERAGE

Checkbox (marqdown)

The system under test will be an microservice extracted from a markdown-based survey site.

The test suite includes the following test markdown files:

  • test/resources/long.md
  • test/resources/survey.md
  • test/resources/upload.md
  • test/resources/variations.md

Starting service can be acheived by installing node modules and running the service.

npm install
node index.js

The rendered survey is accessible on http://localhost:3000/survey/{filename}

Mutation Coverage Approach

Use the following guidelines for implementing your mutation coverage.

  1. Generate snapshots.

Generate your initial baseline snapshots of the test suite.

An example implementation of a headless browser image-based snapshot is provided.

  1. Mutation operators.

Implement the following mutation operations (with examples).

  • Conditional boundary mutations: > => >=, < => <=
  • Incremental mutations: ++j =>j++, i++ => i--
  • Negate conditionals: == => !=, > => <
  • Mutate control flow if => else if
  • Conditional expression mutation && => ||, || => &&
  • Clone return, early Find: return embeddedHtml;, copy and insert in random location of function (before declaration).
  • Non-empty string: "" => "<div>Bug</div>".
  • Constant Replacement: 0 => 3

An example implementation of a mutation operator is provided.

  1. Test harness

Create a test harness that will be sufficient for allowing you to:

  • Start with original version of code.
  • Apply mutation operator.
  • Run service with mutated code.
  • Obtain snapshot of mutated code.
  • Compare snapshots to determine if any difference (using image-based or DOM-based difference)
  1. Mutation coverage

Using your test harness, generate 1000 mutations, randomly applying one mutation operator, in a random applicable location of code.

For each mutation, report the operator applied, the source line changed, and the end result of mutation. Retain the snapshots for inspection by teaching staff.

The mutation coverage is calculated by: any test case fails for mutant / total number of mutations.

  1. Build specification

Update your build.yml with a mutation-coverage build job. Example specification:

setup: 
    ...
jobs:
    ... 
  - name: mutation-coverage
    mutation:
      url: https://github.com/chrisparnin/checkbox.io-micro-preview
      iterations: 1000
      snapshots:
       - http://localhost:3000/survey/long.md
       - http://localhost:3000/survey/upload.md
       - http://localhost:3000/survey/survey.md
       - http://localhost:3000/survey/variations.md

Milestone report

Document the experiences you have in designing and implementing your analysis. This can be described in your submission's README.md.

💯 Evaluation

Your analysis be evaluated based on expressiveness and generalizablity of your build specification, and effectiveness, soundness, and quality of your implmementation.

Your pipeline must remain compatible with existing pipeline build jobs (M1).

  • Mutation operators (30%)
  • Test harness (30%)
  • Snapshot oracle and differencing (20%)
  • Screencast and milestone report. (20%)

Points may be deducted for not following constraints, poor quality of implementation, poor task communication and delegation, or failing to include sufficient detail required to evaluate capabilities.

  • Updating and placing the provided .env file in README.md in root directory of project
  • Running npm install
  • Running node index.js init
  • Running node index.js build itrust-build build.yml
  • Running node index.js build mutation-coverage build.yml

Points will be deducted for non-contributing members, including receiving zero credit.

Submission

Commit your all your work related to your project into your designated repository in the master branch before the final deadline.

When you have finished the submission, create a branch called M2 to hold a snapshot of your work related to this milestone.

Ensure your repository contains:

  • a top-level node.js project supporting the required commands for running your build server.
  • a sample .env file that describes any necessary settings needed for running your pipeline.
  • a README.md, with your report details.
  • a link to screencast that demostrates your analysis.

Due Wednesday, April 13th, before midnight.