[jazzy] Improve the reliability of rosbag2 tests (backport #1796)

[mergify]

I initially started this series trying to track down a rare failing flakey test with test_record__rmw_cyclonedds_cpp. That particular flake seems to be able to happen because sometimes discovery takes longer than we expect, and it is possible that the tests "miss" the first publication. If that's the case, then the rest of the test may fail because it is expecting a certai number of messages. Along the way, we cleanup the tests a bit:

1. Remove the unnecessary wait_until_shutdown method, which was almost exactly the same as wait_for_condition.
2. Slightly revamp how the async spinner works, so it doesn't need to call rclcpp::shutdown.
3. Do a similar revamp for the custom async spinner in rosbag2_tests.
4. Wait for topics to be discovered right after calling recorder->record(). This ensures we can't get into the above situation.

With this series in place, the particular flake of test_record on rmw_cyclonedds_cpp is fixed (or, at least, I can no longer reproduce it). There are still other flakes that can happen under load, but I think fixes for those will have to come separately.

---

This is an automatic backport of pull request #1796 done by Mergify.

[mergify]

Cherry-pick of 2d4d02f has failed:

On branch mergify/bp/jazzy/pr-1796
Your branch is up to date with 'origin/jazzy'.

You are currently cherry-picking commit 2d4d02fd.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   rosbag2_test_common/include/rosbag2_test_common/wait_for.hpp
	modified:   rosbag2_tests/test/rosbag2_tests/test_rosbag2_cpp_get_service_info.cpp
	modified:   rosbag2_transport/test/rosbag2_transport/record_integration_fixture.hpp
	modified:   rosbag2_transport/test/rosbag2_transport/test_keyboard_controls.cpp
	modified:   rosbag2_transport/test/rosbag2_transport/test_record_all_include_unpublished_topics.cpp

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   rosbag2_transport/test/rosbag2_transport/test_record.cpp
	both modified:   rosbag2_transport/test/rosbag2_transport/test_record_all.cpp
	both modified:   rosbag2_transport/test/rosbag2_transport/test_record_all_ignore_leaf_topics.cpp
	both modified:   rosbag2_transport/test/rosbag2_transport/test_record_all_no_discovery.cpp
	both modified:   rosbag2_transport/test/rosbag2_transport/test_record_all_use_sim_time.cpp
	both modified:   rosbag2_transport/test/rosbag2_transport/test_record_regex.cpp

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[MichaelOrlov]

Pulls: #1806
Gist: https://gist.githubusercontent.com/MichaelOrlov/ab10b4f5b37005d0a57f1def84281c46/raw/6808c27bfb04669c663be100567fdb7199d995ca/ros2.repos
BUILD args: --packages-above-and-dependencies rosbag2_test_common rosbag2_tests rosbag2_transport
TEST args: --packages-above rosbag2_test_common rosbag2_tests rosbag2_transport
ROS Distro: jazzy
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/14580

• Linux
• Linux-aarch64
• Linux-rhel
• Windows

[MichaelOrlov]

The redhat build finished with warnings no CONNEXTDDS_DIR nor NDDSHOME specified. However, this is unrelated to this PR. Merging then.