Look into ways to speed up PHPunit tests #507
Comments
|
Another possibility would be to look into whether any of our existing functional browser tests might be able to be converted into Kernel tests: |
|
Just as a baseline... I see your point. |
|
(Anyone else take notes for themselves and others on tickets? Sometimes I leave a novel... let me know if I should do this some other way.) Just one test is about 1/10 the time. Makes sense: And two tests takes about 20% percent of the time. Maybe obvious, but this demonstrates that the testing time scales linearly with the number of full Drupal bootstraps. Makes perfect sense, just helpful to validate since I haven't looked at this test suite yet: $ lando phpunit --group az_demo,az_paragraphs --verbose
.. 2 / 2 (100%)
Time: 3.7 minutes, Memory: 8.00 MB
OK (2 tests, 17 assertions)Changing the group of This is because So Drupal installation state doesn't automatically stay across tests. Not surprising... I might be concerned if it did! |
|
Oops, closed accidentally. |
|
I'm open to hearing more about the test grouping idea but, as implied in the description
this method will increase coupling between what should otherwise be independent tests, which may increase the difficulty of writing tests. I'm curious about the Functional->Kernel test conversion idea and will head there next. It's more in line with the idea of a "testing pyramid". |
|
I haven't gone down the Kernel test route yet, mainly because I think it will make writing tests harder, if any of them are good fits for Kernel tests in the first place. Some are not, I think. I did look into test parallelization using https://github.com/liuggio/fastest, and that seems to be helping, though I only just started down the path of making sure tests don't collide in the database. Instead of 18 minutes, the tests took half that on my 4-core laptop: I'm not sure if this is really a viable approach, and I'm not sure if this will help Probo. I can clean this up soon and attempt a Probo build to see. @joeparsons @tadean or anyone else: Does it seem like this approach could help? |
|
@michaelhagedon |
|
I didn't realize that If someone wants to see how that works, it starts in |
If this runs at all the first time, we'll see how many cores Probo dedicates.
|
Pushed a Probo modification to use 40 cores possible, and 3 minutes run time... encouraging! |
|
Well, first almost-good run failed a filesystem issue. Traced it to I see we're creating a simpletest directory ahead of time in the probo config at |
|
Well, I ran a rebuild and it worked! 3 min 5 sec! But that first failure smells of non-determinism. I do see that the simpletest directory we're creating does have stuff in it during the run, but I'm thinking Drupal also created this one:
|
…and avoid non-determinism. (#507)
|
I decided to preemptively create that other simpletest directory, though I'm not sure how to prove that I need to do that. Next build was just under 3 minutes! |
|
If this is on a good path and we don't find red flags, I'm certain there's some refactoring needed. This new dependency should probably be in the az-quickstart-dev for one thing. Also I have an extra lando command too that runs phpunit this way. |
|
@michaelhagedon We recently added a unit test that uses PHPunit data providers. I saw this in the liuggio/fastest issue queue that makes me wonder if it will be able to run our new test: liuggio/fastest#8 Any idea? |
|
@joeparsons Interesting find! I'm not 100% sure, but I think that issue is referring to parallelizing the run of each individual array element input from the data provider. But in our case, I think it doesn't matter. This new test is hundreds of times faster than the browser tests, so if the inputs from the data provider all run in sequence, that's fine. Here's what happens if I run all the tests now: The new one seems to work fine. I also tried intentionally braking everything in the data provider by adding a character to all the So I think it's OK. Glad you checked! |
|
10 minutes 21 seconds 201 milliseconds. time for a new computer, you need 40 cores |
|
@trackleft I should definitely put 40 cores as a requirement on my next laptop request! |
What is the problem that we want to solve?
The amount of time it takes to run our suite of PHPunit tests is getting pretty long.
During the 2021-02-10 workshop @tadean suggested we might be able to speed up the execution of our tests if we apply the same group name to more of our tests to reduce the number of times Drupal is installed (as long as we don't group any tests that would conflict with each other together).
The text was updated successfully, but these errors were encountered: