Running queue mode locally for troubleshooting with a small subset of tests, but with multiple batches

[technicalpickles]

We are thinking about #200 again, and trying to find ways to reproduce the queue runner locally so we can use memory_profiler on it.

https://docs.knapsackpro.com/ruby/troubleshooting/ outlines the broad steps, but the problems I'm running into with reproducing:

• since this is hitting the API, I'm worried that this is going to pollute our spec's runtime
• I can a KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE with the files to use, but I'm finding it hard to balance getting a small enough batch that it doesn't take too long, but not too big that it takes too long

I think ideally, I'd be able to specify a few batches of specs and run those, rather than fetching the API.

[technicalpickles]

Another reason we'd like this is for when we need to work on our CI rspec formatters, and their interactions with the ones Knapsack Pro adds.

In the sort term, I've been able to kinda fake this by adding an arbitrary loop around

knapsack_pro-ruby/lib/knapsack_pro/runners/queue/rspec_runner.rb

Lines 80 to 122 in ddc8f71

	subset_queue_id = KnapsackPro::Config::EnvGenerator.set_subset_queue_id
	ENV['KNAPSACK_PRO_SUBSET_QUEUE_ID'] = subset_queue_id
	KnapsackPro.tracker.reset!
	KnapsackPro.tracker.set_prerun_tests(test_file_paths)
	KnapsackPro::Hooks::Queue.call_before_subset_queue
	all_test_file_paths += test_file_paths
	cli_args = args + test_file_paths
	ensure_spec_opts_have_rspec_queue_summary_formatter
	options = ::RSpec::Core::ConfigurationOptions.new(cli_args)
	rspec_runner = ::RSpec::Core::Runner.new(options)
	begin
	exit_code = rspec_runner.run($stderr, $stdout)
	exitstatus = exit_code if exit_code != 0
	rescue Exception => exception
	KnapsackPro.logger.error("Having exception when running RSpec: #{exception.inspect}")
	KnapsackPro::Formatters::RSpecQueueSummaryFormatter.print_exit_summary
	KnapsackPro::Hooks::Queue.call_after_subset_queue
	KnapsackPro::Hooks::Queue.call_after_queue
	Kernel.exit(1)
	raise
	else
	if rspec_runner.world.wants_to_quit
	KnapsackPro.logger.warn('RSpec wants to quit.')
	set_terminate_process
	end
	if rspec_runner.world.rspec_is_quitting
	KnapsackPro.logger.warn('RSpec is quitting.')
	set_terminate_process
	end
	printable_args = args_with_seed_option_added_when_viable(args, rspec_runner)
	log_rspec_command(printable_args, test_file_paths, :subset_queue)
	rspec_clear_examples
	KnapsackPro::Hooks::Queue.call_after_subset_queue
	KnapsackPro::Report.save_subset_queue_to_file

[ArturT]

I think ideally, I'd be able to specify a few batches of specs and run those, rather than fetching the API.

Maybe this will be helpful. Here is an example of how I explicitly defined custom tests batches when I wanted to debug something.
https://github.com/KnapsackPro/knapsack_pro-ruby/pull/161/files

[ArturT]

since this is hitting the API, I'm worried that this is going to pollute our spec's runtime

You can keep using the current API token and just use the separate branch name for testing purposes. The only tiny downside could be that you would affect the mean execution time of tests, but taking into account how often you are running CI builds it would be a negligible impact.

[technicalpickles]

Would it be possible to use a different suite (and thus token) for it? That could help isolate the changes.

I was able to apply the idea from https://github.com/KnapsackPro/knapsack_pro-ruby/pull/161/files . I generalized it a little so I could included it in a file w/o having to modify my gems:

module KnapsackPatch
  @@i = 0
  def test_file_paths(args)
    @@i += 1

    case @@i
    when 1
      [
        "spec/a_spec.rb",
        "spec/b_spec.rb",
      ]
    when 2
      [
        "spec/c_spec.rb",
        "spec/d_spec.rb",
      ]
    else
      []
    end
  end
end

KnapsackPro::Runners::Queue::RSpecRunner.prepend(KnapsackPatch)

[ArturT]

Would it be possible to use a different suite (and thus token) for it? That could help isolate the changes.

Yes. You can run at least one CI build for all test files for a new API token to have execution time recorded.

[ArturT]

#200 is closed, so I think we can close this question/issue for now. Feel free to re-open if you need more help.