Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Organised operators in categories #111

Merged
merged 2 commits into from
Apr 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 11 additions & 89 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,25 +27,28 @@ Metacello new
load.
```

## Analysis
## Quick start

```smalltalk
| analysis alive testClasses classesToMutate |
testClasses := { UUIDPrimitivesTest }.
classesToMutate := { UUID . UUIDGenerator }.
"Put here the classes you want to mutate"
classesToMutate := { }.
"Put here the test classes associated with"
testClasses := { }.

analysis := MTAnalysis new
classesToMutate: classesToMutate;
testClasses: testClasses.

analysis run.

"To retrieve the alive mutations"
alive := analysis generalResult aliveMutants.

"To inspect the results"
analysis generalResult inspect.
```

## Wiki

You can find a wiki [here](https://github.com/pharo-contributions/mutalk/wiki) with a lot of information about MuTalk and how it works. It is higly recommended to read it.

---

Original repository: http://www.squeaksource.com/MutationTesting
Expand All @@ -62,85 +65,4 @@ During the 70s, mutation testing emerged as a technique to assess the fault-find

However, so far it is a “brute force” technique that takes too long to provide useful results. This characteristic has forbidden its widespread and practical use regardless the existence of new techniques, such as schema-based mutation and selective mutation. Additionally, there are no mutation testing tools (to our knowledge) that work on meta-circular and dynamic environments, such as Smalltalk, so compile and link time are the current technique's bottleneck.

This Smalltalk-based tool was developed at the University of Buenos Aires (Argentina) in the context of the final thesis work. The tool uses Smalltalk's dynamic and meta-programming facilities to notably reduce the time to get valuable output and help to understand and implement new tests due to its integration with the rest of the environment.

## Wiki

You can find a wiki [here](https://github.com/pharo-contributions/mutalk/wiki).

## FAQ

> How can I run a mutation testing analysis on my packages?

Currently you need to write a script in the playground (shortcut: CTRL + O + W) like the one above.

In the future we plan to improve the UI to allow MuTalk to be used:
* in the Mutation Testing Runner
* within the Class Browser

> What is the difference between each running mode?

We have four modes to run an analysis:
* Mutate All, Run All: it means mutating all your code and then running all tests.
* ```smalltalk
analysis mutantSelectionStrategy: MTAllMutantSelectionStrategy new;
testSelectionStrategy: MTAllTestsMethodsRunningTestSelectionStrategy new.
```
* Mutate All, run Covering: it means mutating all your code but, for each mutated method, running tests that cover it. The result should be, in general, the same than running Mutate All, Run All, but taking less time.
* ```smalltalk
analysis mutantSelectionStrategy: MTAllMutantSelectionStrategy new;
testSelectionStrategy: MTSelectingFromCoverageTestSelectionStrategy new.
```
* Mutate Covered, Run All: it means mutating only code covered by tests and then running all tests.
* ```smalltalk
analysis mutantSelectionStrategy: MTSelectingFromCoverageMutantSelectionStrategy new;
testSelectionStrategy: MTAllTestsMethodsRunningTestSelectionStrategy new.
```
* Mutate Covered, Run Covering: it means mutating covered code and, for each mutated method, running tests that cover it. The result must be, in general, the same than running Mutate Covered, run All, but taking less time.
* ```smalltalk
analysis mutantSelectionStrategy: MTSelectingFromCoverageMutantSelectionStrategy new;
testSelectionStrategy: MTSelectingFromCoverageTestSelectionStrategy new.
```

> What are the options when running an analysis?

There are various options you can play with.
You can change the mutant selection strategy as we presented above, but there are more. You can manually give the methods to be mutated with `MTManualMutatedMethodSelectionStrategy`, or directly give the mutations with `MTManualMutationSelectionStrategy`.
You can also randomize the mutations order with `MTRandomMutantSelectionStrategy`. This strategy relies on another mutant selection strategy (by default it is the all mutant one) to generate the mutations, then it shuffles the collection of mutations. Another option is to use `MTRandomOperatorMutantSelectionStrategy`, which randomly selects a mutant operator then randomly selects a mutation from this operator.

Theses random strategies are especially useful when paired with budgets, in particular the ones on number of mutants. Budgets are meant to impose a limitation on the analysis.
There are 4 types:
* No budget: the analysis will run until finished or until it encounters a bug.
* ```smalltalk
analysis budget: MTFreeBudget new.
```
* Budget on a fixed number of mutants: the analysis will stop when the given number of mutants have been evaluated.
* ```smalltalk
analysis budget: (MTFixedNumberOfMutantsBudget for: 10).
```
* Budget on a percentage of mutants: the analysis will stop when the given percentage of mutants have been evaluated. If the percentage gives a non exact number of mutants (for instance 7.6 mutants), it stops at the rounded up number (here it would be 8 mutants).
* ```smalltalk
analysis budget: (MTPercentageOfMutantsBudget for: 10).
```
* Budget on time: the analysis will run until the given time as passed. If a mutant is being evaluated when the time is up, the analysis continues until this evaluation is finished.
* ```smalltalk
analysis budget: (MTTimeBudget for: 10 seconds).
```

When evaluating a mutant the analysis installs the mutation, runs all tests, and at the first failing test it ends the evaluation, uninstalls the mutation and moves on to the next mutant. However in some cases you need the analysis to run all tests and do not stop at the first failing test.
You can do that in MuTalk with this option:
```smalltalk
analysis doNotStopOnErrorOrFail.
```

> What is the default mode to run mutation testing?

The default parameters are: mutate all, run all, no budget and stop at first fail.

> Does MuTalk replace coverage analysis?

No, it doesn't. It complements coverage. We should first try to write all the tests to achieve a high coverage score on the code we want to mutate. We can then run mutation testing in Mutate Covered, Run Covering mode.

> My image is freezing when running a mutation testing analysis, what can I do?

You can can open the Mutation Testing Runner and then Activate Logging to File. It will generate a file called MutationTestingLog.txt which lets you know which mutant is causing your image to freeze. You can install the mutation by hand, and then debug running your tests. You will probably need to exclude a class or an operator from the analysis.
This Smalltalk-based tool was developed at the University of Buenos Aires (Argentina) in the context of the final thesis work. The tool uses Smalltalk's dynamic and meta-programming facilities to notably reduce the time to get valuable output and help to understand and implement new tests due to its integration with the rest of the environment.
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTAbstractMutantOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTAbstractMutantOperator',
#superclass : 'Object',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Abstract',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Abstract'
}

{ #category : 'accessing' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTAssignmentNullifierOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTAssignmentNullifierOperator',
#superclass : 'MTPredicateBasedMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Other',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Other'
}

{ #category : 'applying' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTEmptyMethodOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTEmptyMethodOperator',
#superclass : 'MTPredicateBasedMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Deletion',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Deletion'
}

{ #category : 'applying' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTLiteralBooleanNegateOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTLiteralBooleanNegateOperator',
#superclass : 'MTPredicateBasedMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Literal manipulation',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Literal manipulation'
}

{ #category : 'applying' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTLiteralIntegersDecreaseOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTLiteralIntegersDecreaseOperator',
#superclass : 'MTPredicateBasedMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Literal manipulation',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Literal manipulation'
}

{ #category : 'applying' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTLiteralIntegersIncreaseOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTLiteralIntegersIncreaseOperator',
#superclass : 'MTPredicateBasedMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Literal manipulation',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Literal manipulation'
}

{ #category : 'applying' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTLiteralIntegersToZeroOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTLiteralIntegersToZeroOperator',
#superclass : 'MTPredicateBasedMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Literal manipulation',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Literal manipulation'
}

{ #category : 'applying' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTLiteralStringChangeOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTLiteralStringChangeOperator',
#superclass : 'MTPredicateBasedMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Literal manipulation',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Literal manipulation'
}

{ #category : 'applying' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTLiteralStringToEmptyOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTLiteralStringToEmptyOperator',
#superclass : 'MTPredicateBasedMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Literal manipulation',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Literal manipulation'
}

{ #category : 'applying' }
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTMessageSendArguments1stNullifierOperator',
#superclass : 'MTMessageSendArgumentsNullifierOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Message sender',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Message sender'
}

{ #category : 'applying' }
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTMessageSendArguments2ndNullifierOperator',
#superclass : 'MTMessageSendArgumentsNullifierOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Message sender',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Message sender'
}

{ #category : 'applying' }
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTMessageSendArguments3rdNullifierOperator',
#superclass : 'MTMessageSendArgumentsNullifierOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Message sender',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Message sender'
}

{ #category : 'applying' }
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTMessageSendArgumentsNullifierOperator',
#superclass : 'MTPredicateBasedMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Message sender',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Message sender'
}

{ #category : 'testing' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTMessageSendToYourselfOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTMessageSendToYourselfOperator',
#superclass : 'MTPredicateBasedMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Message sender',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Message sender'
}

{ #category : 'applying' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTParseRewriterMutantOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTParseRewriterMutantOperator',
#superclass : 'MTAbstractMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Abstract',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Abstract'
}

{ #category : 'testing' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTPredicateBasedMutantOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTPredicateBasedMutantOperator',
#superclass : 'MTAbstractMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Abstract',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Abstract'
}

{ #category : 'testing' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTRemoveAtIfAbsentOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTRemoveAtIfAbsentOperator',
#superclass : 'MTParseRewriterMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Collection operation',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Collection operation'
}

{ #category : 'printing' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTRemoveCaretOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTRemoveCaretOperator',
#superclass : 'MTParseRewriterMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Deletion',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Deletion'
}

{ #category : 'printing' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTRemoveExceptionHandlerOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTRemoveExceptionHandlerOperator',
#superclass : 'MTParseRewriterMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Deletion',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Deletion'
}

{ #category : 'printing' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTRemoveInjectIntoOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTRemoveInjectIntoOperator',
#superclass : 'MTParseRewriterMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Deletion',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Deletion'
}

{ #category : 'printing' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTRemoveNotOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTRemoveNotOperator',
#superclass : 'MTParseRewriterMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Deletion',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Deletion'
}

{ #category : 'printing' }
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTReplaceAndArgumentWithTrueOperator',
#superclass : 'MTParseRewriterMutantOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Logical boolean operation',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Logical boolean operation'
}

{ #category : 'printing' }
Expand Down
4 changes: 2 additions & 2 deletions src/MuTalk-Model/MTReplaceAndReceiverOperator.class.st
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Class {
#name : 'MTReplaceAndReceiverOperator',
#superclass : 'MTReplaceReceiverOperator',
#category : 'MuTalk-Model-Operators',
#category : 'MuTalk-Model-Operators - Logical boolean operation',
#package : 'MuTalk-Model',
#tag : 'Operators'
#tag : 'Operators - Logical boolean operation'
}

{ #category : 'testing' }
Expand Down
Loading
Loading