Memory leak

[ngan]

Based on this: rspec/rspec-core#2987

It looks like rspec has a memory leak when doing multiple runs within the same process. If I’m not mistaken, I think is how knapsack pro operates. We’ve notice our rspec instance taking significantly more memory in CI after switching to knapsack pro. Just wanted to put this issue up for further discussion.

Along the same lines, it would be better if knapsack pro could write custom runner that queries and adds tests on the fly to a single test run. This would make the output more seamless and address the memory issue, if it were the case.

[ArturT]

It looks like rspec has a memory leak when doing multiple runs within the same process. If I’m not mistaken, I think is how knapsack pro operates.

Yes, we call ::RSpec::Core::Runner for each batch of tests fetched from Queue API.

In past, we noticed that if you were to use KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true and run too many RSpec test cases in queue mode it could lead to more memory consumption so that's why we try to run a minimal number of test cases when you use the RSpec split by test examples feature.

Along the same lines, it would be better if knapsack pro could write custom runner that queries and adds tests on the fly to a single test run. This would make the output more seamless and address the memory issue, if it were the case.

Do you have an idea how this could be done other than calling ::RSpec::Core::Runner?

Here is how it's done for Queue Mode now.

After each batch of tests fetched from Queue API we executed the tests with ::RSpec::Core::Runner. After that we clean up some stuff with the rspec_clear_examples method.
It cleans up a few things:

knapsack_pro-ruby/lib/knapsack_pro/runners/queue/rspec_runner.rb

Line 142 in 45b1b16

def self.rspec_clear_examples

You could use after_subset_queue to run a custom code that could do more cleanup as a next step. Maybe we could put there some code from the PR.

KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
  puts 'after_subset_queue - run after the previous subset of tests'
end

The after_subset_queue hook is called after the rspec_clear_examples.

knapsack_pro-ruby/lib/knapsack_pro/runners/queue/rspec_runner.rb

Line 101 in 45b1b16

KnapsackPro::Hooks::Queue.call_after_subset_queue

[ArturT]

We have released version 7.0.0 of the knapsack_pro gem. This version has many improvements in Queue Mode for RSpec.

We have not found any memory leaks. Therefore, I am closing this issue.

Please find the migration steps at the following link:
https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/CHANGELOG.md#700