support for overfitting detection

[monperrus]

we have made great progress on overfitting detection, In the emails sent by Repairnator, the patches should be ranked by overfitting likelihood, and come with an "overfitting" score

• support for ODS (Nov 2019 version)

[monperrus]

waiting for the Coming issue SpoonLabs/coming#125

before making progress here.

[SophieHYe]

Hi @jaxxgu,

Thanks for making a great contribution! The commits look good. I have a few minor comments:

(1) I noticed overfittingScore can not be computed for NPEFix because of invalid filePaths here. From my understanding, the Nopol patches and NPEFix patches share the same buggy file, should be about to apply the patch and compute the score. My major concern is whether the P4J cannot compute all NPEFix patches or just in the test case? It would be appreciated to let us know more information.
(2) We can see the current overfitting ranking score range is from -3190 to 3465 in ascending order. It's OK for the current issue. We will see if we want to have a normalization for the ranking score in the future.
(3) Current we see we only compute the P4J, I am not sure if we still need Sketch4Repair score.
(4) Maybe a bit comment in the test file and removal of the commented out code.

Thank you for your work! It looks fine to me.

[jianguda]

@SophieHYe
(1) All NPEFix patches are not supported right now, as they only use one blank String to represent buggy file path.
(2) It should be unnecessary, as overfitting score is only used in our ranking task.
(3) It makes sense, I will add more choices and keep the default one being P4J.
(4) Done.

[jianguda]

@SophieHYe
I completed the third point.

[jianguda]

For YE's first point:

Right now, we have four kinds of RepairTools, namely AssertFixer, Astor, Nopol and NPE. However, only AssertFixer and Nopol have set valid FilePath value when constructing RepairPatches. Therefore, all RepairPatches generated by Astor and NPE would always get ranked at the end of our sorted list.

Maybe we could create one new issue for that, as it seems not so easy to fix that without getting deep into these tools.

[monperrus]

Sequencer finds overfitting patches in if conditions, exactly like Nopol and Dynamoth, we need to discard them:

add support for branch overfitting (if the patch is in an if condition, check that both the if and the else branches are executed at least once) 

[monperrus]

Decision of today's meeting: when ODS finds several likely correct patches, we rank according to the beam order

[monperrus]

Discussed today, we need to log when a patch is discarded because of overfitting

[nashid]

@monperrus would be interested to know how you detect overfitting.

What the heuristics that are currently used?

Is it documented somewhere in the readme?

[monperrus]

Hi @nashid

We use the technique described in

Automated Classification of Overfitting Patches with Statically Extracted Code Features
http://arxiv.org/pdf/1910.12057

which we've devised with @SophieHYe and @martinezmatias

[monperrus]

Can be considered as done, follow up is #1135