How to run another test as a sub-process?

[jimav]

This may not be a bug, but I'm confounded trying to understand how to do a seemingly-simple thing in a test: Execute a separate, independent test script in a sub-process.

What I actually want is to repeat a large test several times, varying a parameter (passed on the command line). The test to be repeated mucks with globals and can not be simply wrapped in a loop because of undesired carry-over state; so the test must be re-run in a new process each time.

Without using sub-processes, this seems to work:

# t/99_demo.t (no sub-processes)
use Test2::V0;
use Test2::Tools::Subtest qw/subtest_buffered/;

subtest_buffered st1 => sub{ ok(1, "DD"); done_testing; };
subtest_buffered st2 => sub{ for (1..($ARGV[0]//42)) { ok(1) } done_testing; };
done_testing();

prove t/99_demo.t succeeds.

However a similar test using sub-processes does not. It seems like
the "separate" sub-process somehow hooks itself into the parent process's test harness, and that doesn't play nicely.

# t/100_demo.t
use Test2::V0;
use Test2::Tools::Subtest qw/subtest_buffered/;

subtest_buffered st1 => sub {
  my $wstat = system $^X, '-wE', 'use Test2::V0; for (1..$ARGV[0]//1) { ok(1) } done_testing();', 42;
  ok($wstat==0, "st1 succeeded");
  done_testing();
};
done_testing();

prove 1/100_demo.t says

t/100_demo.t .. All 42 subtests passed 

Test Summary Report
-------------------
t/100_demo.t (Wstat: 0 Tests: 43 Failed: 1)
  Failed test:  1
  Parse errors: Plan (1..42) must be at the beginning or end of the TAP output
                Tests out of sequence.  Found (1) but expected (43)
                More than one plan found in TAP output
                Bad plan.  You planned 42 tests but ran 43.

By the way, I tried just running the sub-processes without using subtest_buffered and got similar TAPish wierdness.

I read about Test2::API::intercept but couldn't figure out how to re-inject the captured events into the main test so success/failure was properly recorded. I figured I should not try to decode the event structures myself.

Any guidance would be greatly appreciated!

[rabbiveesh]

Have you taken a look at Test2::Tools::Spec? You can have the test fix up state in before blocks. Though in general, when it's hard to test like this, the problem is your code, not the test framework

…

On Sun, May 14, 2023, 06:46 Jim Avera ***@***.***> wrote: This may not be a bug, but I'm confounded trying to understand how to do a seemingly-simple thing in a test: Execute a separate, independent test script in a sub-process. What I actually want is to repeat a large test several times, varying a parameter (passed on the command line). The test to be repeated mucks with globals and can not be simply wrapped in a loop because of undesired carry-over state; so the test must be re-run in a new process each time. Without using sub-processes, this seems to work: # t/99_demo.t (no sub-processes) use Test2::V0; use Test2::Tools::Subtest qw/subtest_buffered/; subtest_buffered st1 => sub{ ok(1, "DD"); done_testing; }; subtest_buffered st2 => sub{ for (1..($ARGV[0]//42)) { ok(1) } done_testing; }; done_testing(); prove t/99_demo.t succeeds. However a similar test using sub-processes does not. It seems like the "separate" sub-process somehow hooks itself into the parent process's test harness, and that doesn't play nicely. # t/100_demo.t use Test2::V0; use Test2::Tools::Subtest qw/subtest_buffered/; subtest_buffered st1 => sub { my $wstat = system $^X, '-wE', 'use Test2::V0; for (1..$ARGV[0]//1) { ok(1) } done_testing();', 42; ok($wstat==0, "st1 succeeded"); done_testing(); }; done_testing(); prove 1/100_demo.t says t/100_demo.t .. All 42 subtests passed Test Summary Report ------------------- t/100_demo.t (Wstat: 0 Tests: 43 Failed: 1) Failed test: 1 Parse errors: Plan (1..42) must be at the beginning or end of the TAP output Tests out of sequence. Found (1) but expected (43) More than one plan found in TAP output Bad plan. You planned 42 tests but ran 43. By the way, I tried just running the sub-processes without using subtest_buffered and got similar TAPish wierdness. I read about Test2::API::intercept but couldn't figure out how to re-inject the captured events into the main test so success/failure was properly recorded. I figured I should not try to decode the event structures myself. Any guidance would be greatly appreciated! — Reply to this email directly, view it on GitHub <#934>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFURPKRIKDZ7JO4T2373TE3XGBISNANCNFSM6AAAAAAYA4QFHE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jimav]

Have you taken a look at Test2::Tools::Spec?
You can have the test fix up state in before blocks.

Hi rabbiveesh,
What is meant by "fix up state"? What exactly is wrong with the test state and what needs to be fixed? I'm happy to read docs if you point me.

Merely running the subordinate test script in an 'iso' block (supposedly completely isolated) still causes errors:

# t/103_demo.t
use Test2::V0;
use Test2::Tools::Spec;

describe foo => sub {
  tests exttest => { iso => 1 }, sub {
    my $wstat = system $^X, '-wE', 'use Test2::V0; ok(1); done_testing();';
    ok($wstat==0, "st1 succeeded");
  };
};
done_testing();

prove t/103_demo.t says:

t/103_demo.t .. All 1 subtests passed 

Test Summary Report
-------------------
t/103_demo.t (Wstat: 0 Tests: 2 Failed: 1)
  Failed test:  1
  Parse errors: Plan (1..1) must be at the beginning or end of the TAP output
                Tests out of sequence.  Found (1) but expected (2)
                More than one plan found in TAP output
                Bad plan.  You planned 1 tests but ran 2.

It seems like events from the "isolated" subtest are corrupting the outer test's state.

[exodist]

The problem here is that you loose internal state when you launch your sub-process.

When you call system($^X, ...) it forks a new process, then replaces the process with exec. At this time the new process has no context from the first process. It does not know that tests have started, it does not know that it is in a subtest. It is just a completely new perl process loading a new instance of the test2 framework and making assertions.

To do what you want you need to change your approach. Instead of using system() you need to use fork(). Then in the new process use 'do()' or 'require' to run the test file you want in the subtest.

Also to make all this work you need to load Test2::IPC first thing in the parent process to make the different process talk to each-other.

Currently there is no way to convey the existing state (or fact you are in a subtest) to a child process except by forking and maintaining state, IE not calling exec() or system().

What you want is theoretically possible to implement, possibly with env vars, but it does not exist now, and is not trivial.

[jimav]

On 5/14/23 3:02 PM, Chad Granum wrote: To do what you want you need to change your approach. Instead of using system() you need to use fork(). Then in the new process use 'do()' or 'require' to run the test file you want in the subtest.

Ok, thanks. That is unlikely to work on Windows because fork() is emulated by a new thread running a copy of the Perl interpreter (a "pseudo process").  This currently has awful side-effects such as running all END{} blocks and DESTROY methods whenever a "forked" pseudo-process exits (the emulation does not implement a way for a "child"'s Perl state to silently vanish the way it does in Unix when 'exec' is called).

[sanko]

I'm trying to run parts of my test suite under valgrind so fork or threads won't work for me either. Before I spend too much time trying to figure out a way to share state though Test2::IPC::Driver::File objects pointing at the same directory or something, did you ever get this up and running, @jimav?

[jimav]

@sanko - no, I ended up replacing all the Test2 calls in the 'subtest' with code which just dies if the answer is wrong. Then the parent process calls Test2's "fail()" if the exit status is non-zero. Crude but it let me move on.