Providing parallel mutation runs

[nathanrpage97]

Hey, I've been looking into the project as a I think the idea of mutation testing is interesting.

When I went around and tested it on the rich I began to notice how slow it is. So digging into I've seen that it can only run one mutation at a time.

What is the Issue

This limit comes in as you modify the file that is being run per instance. Forcing only one runtime available.

Proposed Solution

I've created a modified version of mutmut that allows parallelization.

A SourceLoader to allow runtime modification to the file before being imported-- without writing to disk. It then uses a ThreadPoolExecutor to spawn 10 workers (could be later made to be user-set) to run the mutation tests.

Results

The showcase folder shows a temporary example with the rich library.
Run times on my local machine went from ~65 min -> ~15 min.

Limitations

It currently does not work with hammett or unittest as it requires being able to plugin to the test before it is ran.
This problem can be solved for by tweaking hammett to provide arguments to the pytest hooks it loads as part of the Config object.

Other notes

• No longer writing bak files or creating temp directories to run mutmut from

[amirouche]

@nathanrpage97 I cloned your branch, rebased and it seems to still work.

There is few things to improve, in fact I can only think about making the number of threads configurable.

Do you mind doing a PR, or can I use your branch as the basis of PR?

[nathanrpage97]

You will want to look at ThreadPoolExecutor max_workers. I’ve done a draft PR, but I think there are a few things that need to be considered before then.

[nathanrpage97]

Feel free to post an updated PR, I won’t be able to work on it for a while.

[amirouche]

It is required to delete pyc cache and pass PYTHONDONTWRITEBYTECODE=1 in order to make the on the fly file rewriting mechanic to work.

Something like the following the shell:

find . -name=__pycache__ -type -d | xargs rm -rf
PYTHONDONTWRITEBYTECODE=1 mutmut

ref: #195

[nathanrpage97]

PYTHONDONTWRITEBYTECODE can probably be set during the initialization before spawning the additional python processes. That way a user doesn't have to set it manually.

[nathanrpage97]

I think some more importlib hooks will need to be used to ignore any pre-generated *.pyc file.

[amirouche]

Here is the code I use to apply the mutation (it still works with py3.9):

    with open(path) as f:
        source = f.read()

    patched = patch(diff, source)

    import imp

    components = path[:-3].split('/')
    log.trace(components)
    while components:
        for pythonpath in sys.path:
            filepath = os.path.join(pythonpath, '/'.join(components))
            filepath += ".py"
            ok = os.path.exists(filepath)
            if ok:
                module_path = '.'.join(components)
                break
        else:
            components.pop()
            continue
        break
    if module_path is None:
        raise Exception("sys.path oops!")
    log.warning(module_path)

    patched_module = imp.new_module(module_path)
    try:
        exec(patched, patched_module.__dict__)
    except Exception:
        # TODO: syntaxerror, do not produce those mutations
        exec('', patched_module.__dict__)

    sys.modules[module_path] = patched_module

Then I run pytest with the following code:

    if timeout:
        command.insert(0, "timeout {}".format(timeout))

    command.insert(0, "PYTHONDONTWRITEBYTECODE=1")

    if silent and not os.environ.get("DEBUG"):
        command.append("> /dev/null 2>&1")

    os.system(" ".join(command))

I rewrote most of mutmut, I did a release of the fork at https://pypi.org/project/mutation/ (also see the review of mutmut).

[zadigus]

How close are we to getting parallel runs in mutmut?

[boxed]

@zadigus The mutmut3-poc branch has it: https://github.com/boxed/mutmut/blob/mutmut3-poc/mutmut3.py

Not only parallel runs, but also mutation schemata (so it doesn't mutate files in place), and a fork model (greatly speeding up the entire process).

I just have a job that takes all my time, plus other hobby projects and a family so I don't really have the time to work on it much.

[amirouche]

I forked mutmut @ https://github.com/amirouche/python-mutation.

[boxed]

@amirouche That's not a fork of mutmut as far as I can tell. And if it was, you'd be violating the license agreement.

[amirouche]

I fixed the license. Anyway, the code is much different. Tho, I did change the license in the hope of opening a conversation, and exchange.

[boxed]

I just released mutmut 3, which is a big rewrite. I believe this issue no longer applies anymore. Feel free to reopen it if it still exists.