Gremlins can't detect coverage

[th12br]

Hi!

My project (on a Windows machine) has a pretty good test coverage, but gremlins can't detect any tests.

> go test ./...
runs all tests properly

> gremlins unleash
in the same directory runs for a while and comes back with:

Mutation testing completed in 421 milliseconds 370 microseconds
Killed: 0, Lived: 0, Not covered: 551
Timed out: 0, Not viable: 0, Skipped: 0
Test efficacy: 0.00%
Mutator coverage: 0.00%

The configuration is:

silent: false
unleash:
integration: false
dry-run: false
tags: ''
output: 'output.json'
workers: 1
test-cpu: 0
timeout-coefficient: 0
threshold:
efficacy: 0
mutant-coverage: 0

mutants:
arithmetic-base:
enabled: true
conditionals-boundary:
enabled: true
conditionals-negation:
enabled: true
increment-decrement:
enabled: true
invert-assignments:
enabled: false
invert-bitwise:
enabled: false
invert-bwassign:
enabled: false
invert-negatives:
enabled: true
invert-logical:
enabled: false
invert-loopctrl:
enabled: false
remove-self-assignments:
enabled: false

I checked Gremlins' source code but couldn't find any hint.

Cheers,
Thomas

[k3rn31]

Hi,
this should be investigated, it is quite strange.
Gremlins has always had problems with Windows. At some point, we also considered dropping Windows support.
Understanding and fixing this will take some time because none of us has access to a Windows machine.

[th12br]

Hi again,

I checked out Gremlins at let it run on itself with

go run cmd/gremlins/main.go unleash --workers 1

The first problem is on line 28 in profile.go:

blocks, ok := p[pos.Filename]

Profile holds filenames with backslashes on Windows while pos.Filename comes in Linux style.
A simple fix is to change line 28 to

blocks, ok := p[filepath.FromSlash(pos.Filename)]

This change lets the first test "it recognizes CONDITIONAL_BOUNDARY with GTR" in engine_test.go fail for an unclear reason.

The next problem is, that now the process runs into

ERROR: failed to gather coverage: impossible to executeCoverage coverage: exit status 1

Commenting out line 104 in coverage.go (you might consider to remove lines 103-105 completely) shows:

Mutation testing completed in 2 seconds 417 milliseconds
Killed: 0, Lived: 0, Not covered: 10
Timed out: 175, Not viable: 0, Skipped: 0
Test efficacy: 0.00%
Mutator coverage: 0.00%
ERROR: below efficacy-threshold

This one I don't understand yet.

[th12br]

Some more findings:

To forget about the timeouts for now I just changed lines 198 and 199 in executor.go from

ctx, cancel := context.WithTimeout(context.Background(), m.testExecutionTime)
defer cancel()

to

ctx := context.WithoutCancel(context.Background())

With one worker only, it run quite fast:
> go run cmd/gremlins/main.go unleash --workers 1 --test-cpu 4

Mutation testing completed in 10 seconds 302 milliseconds
Killed: 174, Lived: 0, Not covered: 10
Timed out: 0, Not viable: 0, Skipped: 0
Test efficacy: 100.00%
Mutator coverage: 94.57%

With standard settings it takes much longer and some mutants survive:
> go run cmd/gremlins/main.go unleash

Mutation testing completed in 3 minutes 11 seconds
Killed: 161, Lived: 13, Not covered: 10
Timed out: 0, Not viable: 0, Skipped: 0
Test efficacy: 92.53%
Mutator coverage: 94.57%

Does it make any sense to you?

I still cannot find out why "it recognizes CONDITIONAL_BOUNDARY with GTR" in engine_test.go fails due to my very first change. Does your machine run into the same problem?

[th12br]

Can anybody have a look at it please. I think I can make Gremlins ready for Windows, but I need your feedback.

[rusinikita]

Can you create a fork and PR draft? I'll take a look.

[th12br]

Done.

[rusinikita]

@th12br I left some comments. You can try my suggestion and see what happens