Reqnroll
Replies: 5 comments 4 replies
-
|
I have tried to reproduce this with our basic sample project: https://github.com/reqnroll/Reqnroll.ExploratoryTestProjects/tree/main/ParallelExecution/ReqnrollCalculator.Specs.XUnitFw, but could not. In that sample there are 3 feature files and if I set the "maxParallelThreads" to "2" from json (or from code) it completes the two features is starts with and only after it continues with the third one. Could you please try to reproduce it on a minimalistic sample project that you can share? In addition to that, you can try to change the config from code, instead of json to check if it has any impact (see sample here). Also you can try to make some logging to a file, like what I do here. This produces a log file where you can track the execution order. In my example it produced: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @gasparnagy thanks for your answer, I tried to log the order of the test, in this case I run 3 feature files with the maxParallelThreads set to 2 also, the Auth has 5 tests scenario outlines for a total of tests, the Products has 7 tests scenario outlines for a total of 7 tests and the FileProcessor has 3 scenario for a total of 3 tests and also I have a [ScenarioDependencies] tag. Looking at the logs I see that it starts running 2 tests but then it continues with 3 and the after feature log is run at the end in all iterations I've tried.
|
Beta Was this translation helpful? Give feedback.
-
|
@llbonet14 Are you sure you have configured xUnit worker counts correcty? Because from your trace, it seems that there are 3 workers running in parallel (the number after the timestamp is the worker id): So it seems that worker 4 deals with Please try to set the config from code (see example) to make sure the problem is not with the xUnit JSON config. |
Beta Was this translation helpful? Give feedback.
-
|
I use this file
And this is the configuration I use with specflow, the strange thing is that worker 4 does not start until one test of other workers have finished. As you can see worker 2 and 3 start at the same time and after 48 seconds the other worker starts. Also the all the AfterFeature are run at the same time after all test from all featureFiles have finished, but some features have finished before |
Beta Was this translation helpful? Give feedback.
-
This is not true. If you check the beginning of your log, you can see that the first "AfterScenario" (so the first finished test) is at 1:16, while worker 4 starts already at 0:48.
This is a behavior that Reqnroll introduced. It delays the execution of the AfterFeature until a test from another feature has been scheduled to the same worker. So this is by design.
I don't know. From your log it seems that there are 3 workers running: for example at 1:00 of your execution time the scenarios
have all started and none of them finished yet. As you see my trace above, it worked for me, so there must be something special in your setup that causes this. You should somehow try to figure out what it is, so that we can help. This is why I suggested to test the code-based configuration, but maybe there is something else too. |
Beta Was this translation helpful? Give feedback.
-
|
I downloaded your test project |
Beta Was this translation helpful? Give feedback.
-
|
Could this be related to xunit/xunit#2003? |
Beta Was this translation helpful? Give feedback.
-
|
That is a good spot @obligaron! @llbonet14 Could you please confirm that the changes you made on the sample project were related to async execution? Could you please just share all changes you made? |
Beta Was this translation helpful? Give feedback.
-
|
Here I have the changes but basically I updated the xunit nuget and add some more feature files seting the max parallel to 2. The feature files are copies from the existing ones just changing the names. I did not change any test logic, that's when I thought it could be related with xunit because they had some break changes in version 2.8 I had to adapt my project to continue working with the
|
Beta Was this translation helpful? Give feedback.
-
I'm migrating my specflow test project to reqnroll, I've been able to fix some issues I've found, but there's one issue I'm not able to fix. I used to run the feature files in parallel in specflow where the maxParallelThreads set the maximum number of feature files to run in parallel, so if set to 2 there where 2 drivers running all the scenarios from that feature file and no other feature file started until one was finished. But what currently happens is that at the start the max threats are respected and only that number of feature files are run, but as soon as one of the scenarios has finished (even though there are many more in that feature) another feature has started and it continues without respecting the maxParallelThreads. This is my xunit config file (which worked with specflow)
{ "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json", "parallelizeAssembly": false, "parallelizeTestCollections": true, "maxParallelThreads": 3, "parallelAlgorithm": "aggressive", "diagnosticMessages": true, "internalDiagnosticMessages": true }This is how it starts
This is how continues
This is my VS Test explorer with the feature files
Beta Was this translation helpful? Give feedback.
All reactions