7903519 : jtreg/jtharness is missing features for basic crash testing

[andrlos]

provides SPI for enabling external status transformations of failed tests

this is a continuation of efforts after openjdk/jtharness#59

Requires newest jtharness build (not even tagged yet) that includes above mentioned change to be compiled succesfully

The main idea is to provide a unified StatusTransformer interface, that can be externally implemented by users and added to a classpath in a separate jar to allow modifications of test execution status based on some elementary analysis. This can be easily used for crashtesting (filtering out only tests with jvm crashes).

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• CODETOOLS-7903519: jtreg/jtharness is missing features for basic crash testing (Enhancement - P3)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jtreg.git pull/235/head:pull/235
$ git checkout pull/235

Update a local copy of the PR:
$ git checkout pull/235
$ git pull https://git.openjdk.org/jtreg.git pull/235/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 235

View PR using the GUI difftool:
$ git pr show -t 235

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jtreg/pull/235.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back andrlos! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[mlbridge]

Webrevs

• 00: Full (bc21f0fe)

[sormuras]

A test status listener (in the sense of JUnit's TestWatcher) would be okay-ish, but a general test status transforming SPI is too intrusive and intransparent ... and also exposing an internal field of jtharness's Script class.

In the light of the above, I tend to close this PR.

What would an implementation handling "crashtesting" look like? Do you have a draft for it?

[andrlos]

@sormuras feast your eyes on the code below :D

public class CrashOnlyStatusTransformer implements StatusTransformer {
        @Override
        public Status transform(Status originalStatus, TestDescription td) {
            Status newStatus = originalStatus;
            if(originalStatus.getType() == Status.FAILED && ! this.didCrash(td)){
                newStatus = new Status(Status.PASSED, "Just a regular failure.");
            }
            return newStatus;
        }

        private boolean didCrash(TestDescription td){
            Pattern pattern = Pattern.compile("^hs_err_pid(\\d+)\\.log");
            List<String> hs_errs = Arrays.stream(td.getDir().list()).filter(pattern.asPredicate()).collect(Collectors.toList());
            return !hs_errs.isEmpty();
    }
}

this is an approach that we use for crashtesting with debug jdk builds to separate crashes from regular failures

[jaikiran]

Hello @andrlos, looking at that code you pasted:

if(originalStatus.getType() == Status.FAILED && ! this.didCrash(td)){
    newStatus = new Status(Status.PASSED, "Just a regular failure.");
}

It looks odd to be marking a failed test as successful. Furthermore, doesn't a crashed JVM result in test status to be Status.ERROR?

[sormuras]

[...] this is an approach that we use for crashtesting with debug jdk builds to separate crashes from regular failures

And sometimes tests do produce hs_err_pid.log files and expect/assert them; marking those tests as PASSED, right?

Where (console log, web-view, ...) do you check for such crashes? Manually or with tool/script support?

Isn't it possible to implement/apply an after-the-fact filter that doesn't rewrite actual run results?

[andrlos]

@sormuras it proved to be much harder to filter them after, as the hs_err_pid log is not being copied, plus we have many tests that run via a shell script, so a generic jvm watcher is not an option as would be the option with jvm agent that we use for junit testing.. we need to cover cases, where jvm crashes even tho it has been run via a shell script (or multiple levels of scripts) and watching for hs_err_pid.log proved to be the most reliable and universal approach.

[andrlos]

@jaikiran
the idea is that if the crashed test is resulting in an error, we still want it to be reported, so those remain unchanged.. if the test has crashed but is reported as failure (usually shell scripts executing JVM) then we want them to stay as they are.. same with the test that test are reported as passed but had a crash occur during execution (we check how the jvm is behaving after crash - e.g. give correct links to report the jvm crash etc..) but when it is reported as failed without the crash happening, we want to ignore those for debug testing... but the code is just a very primitive demo of capabilities of this serviceLookup hook..

[andrlos]

@sormuras @jaikiran I would like to also point out, since Junit has been already mentioned here, that JUnit since JUnit4 provides a feature called TestRule where you can implement a custom testRule, that allows the user to modify behavior of any test.. It can be used in a similar scenario, by implementing a TestRule that ignores any regular exceptions, thus only reporting JVM crashes as failures. https://github.com/junit-team/junit4/blob/HEAD/doc/ReleaseNotes4.9.md

[andrlos]

I believe that this approach is pretty minimalistic when all the capabilities for the user are taken into account and presents almost no maintenance overhead. Plus the user of the jtreg project is someone who should know what he is doing and giving him tools to peek inside and modify the test evaluation behavior is nothing bad IMO. Default behavior of jtreg release is also not affected by this change.

[bridgekeeper]

@andrlos This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply add a new comment to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

[andrlos]

@jaikiran @sormuras WDYT?

[bridgekeeper]

@andrlos This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply add a new comment to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

[andrlos]

@jaikiran @sormuras I believe that I answered all your questions and addressed all your comments and currently I am still waiting for a constructive discussion to take place.

[sormuras]

With jtreg 7.5.1 being released the other day and soon to be made default in JDK mainline, I will pick up the thread here soon again. Sorry for the delay.

[jaikiran]

Hello @andrlos, sorry to keep you waiting. I don't have a strong opinion or objection to this change. At the same time, I don't have enough knowledge of this code to approve this change. So other than asking some basic questions (which I did), I don't feel qualified enough to officially review this proposed enhancement.

[sormuras]

The use-case doesn't carry the weight of introducing and maintaining a SPI for arbitrary and distant test result mangling in the inards of jtreg. The aforementioned JUnit TestRule is a good example for a non-distant (think: test-local) way to influnce the result of a test in an understandable way.

In the light of the discussion above, I won't approve this pull request.

Let's discuss other approaches to achieve the goals expressed in https://bugs.openjdk.org/browse/CODETOOLS-7903519

Update 2025-04-29: Let's discuss other approaches to achieve the goals expressed in https://bugs.openjdk.org/browse/CODETOOLS-7903519 here in this PR, and create a new feature request issue and PR on demand.

[andrlos]

Hi @sormuras do you have any ideas about possible approach? It seems that I either lost my openjdk login or I never had one to start with.. I will try and get it while you can start the discussion there. Thanks!

[judovana]

@andrlos please refresh/negotiate your openjdk jira system login. Lets continue the discussion here as sormuras required. Thaxn all!

[bridgekeeper]

@andrlos This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply add a new comment to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

[sormuras]

As this thread already contains most of the related discussion, related topics, and affected people, let's continue here. Traces are recorded automatically to https://mail.openjdk.org/mailman/listinfo/jtreg-dev 's archives.

[andrlos]

Okay, so the main idea would be to integrate a mechanism similar to JUNIT's TestRule.. Do you have any suggestions of where you would see such a feature? I already proposed a way to integrate something similar into the RegressionScript code and further propagate it into the underlying JTHarness.. I am afraid that we can't really avoid propagation into jtharness to get that local non-distant approach.

[sormuras]

What I still don't get is that you want to address a global goal "basic crash testing" with a very local mechanism, like something similar to JUnit 4 "TestRule".

I think a solution lies in between your lines:

it proved to be much harder to filter them after, as the hs_err_pid log is not being copied, plus we have many tests that run via a shell script, so a generic jvm watcher is not an option as would be the option with jvm agent that we use for junit testing.. we need to cover cases, where jvm crashes even tho it has been run via a shell script (or multiple levels of scripts) and watching for hs_err_pid.log proved to be the most reliable and universal approach.

So, what about making sure that jtreg copies hs_err_pid files correctly?