Enable time-based load test mechanism in openjdk-systemtest load tests #396
Conversation
|
Tests:
|
Mesbah-Alam
changed the title
| if (isTimeBasedLoadTest) { | ||
| loadTestInvocation = loadTestInvocation.setTimeLimit(timeLimit); // If it's a time based test, stop execution after given time duration | ||
| } | ||
|
|
||
| loadTestInvocation = loadTestInvocation.setAbortIfOutOfMemory(false) | ||
| .addSuite("classloading") | ||
| .setSuiteThreadCount(cpuCount - 1, 10) | ||
| .setSuiteInventory(inventoryFile) | ||
| .setSuiteNumTests(totalTests * testCountMultiplier) | ||
| .setSuiteRandomSelection(); | ||
| .setSuiteInventory(inventoryFile); | ||
|
|
||
| if (!isTimeBasedLoadTest) { | ||
| loadTestInvocation = loadTestInvocation.setSuiteNumTests(totalTests * testCountMultiplier); | ||
| } |
There was a problem hiding this comment.
Is it possible to have a simpler logic?
if (isTimeBasedLoadTest) {
...
} else {
...
}
Same for the rest of PR
There was a problem hiding this comment.
This is done the way it is because STF framework enforces a particular sequence in which we can add methods in a process definition. For example, in the code snippet above, if instead we did the following:
if (isTimeBasedLoadTest) {
loadTestInvocation = loadTestInvocation.setTimeLimit(timeLimit); // If it's a time based test, stop execution after given time duration
} else {
loadTestInvocation = loadTestInvocation.setSuiteNumTests(totalTests * testCountMultiplier);
}
loadTestInvocation = loadTestInvocation.setAbortIfOutOfMemory(false)
.addSuite("classloading")
.setSuiteThreadCount(cpuCount - 1, 10)
.setSuiteInventory(inventoryFile);
We will have an error like this:
GEN 16:00:20.802 - Using Mode NoOptions. Values = ''
GEN stderr Exception in thread "main" net.adoptopenjdk.stf.StfException: Java invocation built out of sequence. LOAD_TEST_ARGS cannot be set after SUITE
GEN stderr at net.adoptopenjdk.stf.processes.definitions.LoadTestProcessDefinition.checkAndUpdateLevel(LoadTestProcessDefinition.java:784)
GEN stderr at net.adoptopenjdk.stf.processes.definitions.LoadTestProcessDefinition.setAbortIfOutOfMemory(LoadTestProcessDefinition.java:297)
GEN stderr at net.adoptopenjdk.stf.ClassloadingLoadTest.execute(ClassloadingLoadTest.java:92)
GEN stderr at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
GEN stderr at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
GEN stderr at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
GEN stderr at java.base/java.lang.reflect.Method.invoke(Method.java:566)
GEN stderr at net.adoptopenjdk.stf.runner.StfRunner.runStage(StfRunner.java:549)
GEN stderr at net.adoptopenjdk.stf.runner.StfRunner.executeTest(StfRunner.java:297)
GEN stderr at net.adoptopenjdk.stf.runner.StfRunner.main(StfRunner.java:69)
Generation failed
Please see here.
There was a problem hiding this comment.
I guess, we can have a separate issue to investigate if we need to revisit this feature in STF.
There was a problem hiding this comment.
Yes, we should. Please open an issue in STF. Thanks
Also, could you try one last thing? Could you try to move if-else code after setAbortIfOutOfMemory but before addSuite? According to STF code, setAbortIfOutOfMemory is set to LOAD_TEST_ARGS , setSuiteNumTests is set to SUITE , setTimeLimit is set to LOAD_TEST_ARGS and addSuite is set to SUITE .
There was a problem hiding this comment.
move if-else code after setAbortIfOutOfMemory but before addSuite?
This had the same error as before. But it worked if I moved the if-else after addSuite and before setSuiteThreadCount :
LoadTestProcessDefinition loadTestInvocation = test.createLoadTestSpecification()
.addJvmOption("-Djava.classloading.dir=" + notonclasspathDirRoot) // Expose the bin_notonclasspath root directory to the deadlock test
.addJvmOption("-Djava.version.number=" + javaVersion)
.addModules(modulesAdd)
.addPrereqJarToClasspath(JavaProcessDefinition.JarId.JUNIT)
.addPrereqJarToClasspath(JavaProcessDefinition.JarId.HAMCREST)
.addProjectToClasspath("openjdk.test.classloading")
.setAbortIfOutOfMemory(false)
.addSuite("classloading");
if (isTimeBasedLoadTest) {
loadTestInvocation = loadTestInvocation.setTimeLimit(timeLimit); // If it's a time based test, stop execution after given time duration
} else {
loadTestInvocation = loadTestInvocation.setSuiteNumTests(totalTests * testCountMultiplier);
}
loadTestInvocation = loadTestInvocation.setSuiteThreadCount(cpuCount - 1, 10)
.setSuiteInventory(inventoryFile)
.setSuiteRandomSelection();
So, this simplifies the logic too. I will make similar update in the rest of the test classes.
There was a problem hiding this comment.
New STF issue raised: adoptium/STF#101
There was a problem hiding this comment.
Just to recap conversation with Lan about the above issue over slack:
I tried the change proposed above locally, and the if-else doesn't really work for both the scenarios when a timeLimit is provided (time-based test) and when it's no (iteration based test). Meaning, one particular sequence works when isTimeBasedLoadTest==true, and a different sequence is expected by STF when isTimeBasedLoadTest=false. Hence we have decided to stick to what the original change was.
Further discussion to commence in adoptium/STF#101
| loadTestInvocation = loadTestInvocation.setSuiteNumTests(totalTests * testCountMultiplier); | ||
| } | ||
|
|
||
| loadTestInvocation = loadTestInvocation.setSuiteRandomSelection(); |
There was a problem hiding this comment.
Can we set setSuiteRandomSelection , setAbortIfOutOfMemory , and others together in line 78? Same for the rest of PR
Mesbah-Alam
force-pushed
the
add-timebased-tests
branch
from
68838eb
to
3eea79b
Compare
|
Updated global default timeout value to 1h. Started a sanity.system build on hotspot / jdk11: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/6132/ |
Signed-off-by: [email protected] [email protected]