add spawn cuda process testing #535

sichu2023 · 2024-12-16T18:06:11Z

No description provided.

jstjohn · 2024-12-16T18:12:02Z

sub-packages/bionemo-testing/tests/bionemo/testing/test_megatron_parallel_state_utils.py

+        assert tensor.item() == world_size * (world_size + 1) / 2
+
+
+@pytest.mark.skipif(torch.cuda.device_count() > 1, reason=f"Requires 2 devices but got {torch.cuda.device_count()}")


shouldn't this be skipif device_count() < 2?

jstjohn · 2024-12-16T18:14:52Z

sub-packages/bionemo-testing/tests/bionemo/testing/test_megatron_parallel_state_utils.py

+
+
+def _test_all_reduce_sum(rank: int, world_size: int):
+    with dist_environment(rank=rank, world_size=world_size):


Would be nice to eventually not need any kind of mocking etc, and have some kind of vanilla comparison of results through some kind of standard lightning etc function that's distributed with different levels of different kinds of parallelism asserting that results are equal to the base case (DDP only).

add spawn cuda process testing

4b605ed

jstjohn reviewed Dec 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add spawn cuda process testing #535

add spawn cuda process testing #535

sichu2023 commented Dec 16, 2024

jstjohn Dec 16, 2024

jstjohn Dec 16, 2024

		assert tensor.item() == world_size * (world_size + 1) / 2


		@pytest.mark.skipif(torch.cuda.device_count() > 1, reason=f"Requires 2 devices but got {torch.cuda.device_count()}")



		def _test_all_reduce_sum(rank: int, world_size: int):
		with dist_environment(rank=rank, world_size=world_size):

add spawn cuda process testing #535

Are you sure you want to change the base?

add spawn cuda process testing #535

Conversation

sichu2023 commented Dec 16, 2024

jstjohn Dec 16, 2024

Choose a reason for hiding this comment

jstjohn Dec 16, 2024

Choose a reason for hiding this comment