support pytorch lightning 1.7 #196

JiahaoYao · 2022-08-11T23:44:52Z

fix the issue [Code] add pytorch-lightning compatibility for 1.7.x #194

setup.py

sxjscience · 2022-08-13T03:32:18Z

Checking the CI failure about

E       TypeError: __init__() got an unexpected keyword argument 'checkpoint_callback'

It might be related to these two lines:

ray_lightning/ray_lightning/tests/utils.py

Line 231 in 13468d7

checkpoint_callback=checkpoint_callback,
ray_lightning/ray_lightning/tests/test_tune.py

Line 34 in 13468d7

checkpoint_callback=False,

JiahaoYao · 2022-08-15T23:18:33Z

interesting, the other tests are hanging but they are good on the servers.

JiahaoYao · 2022-08-16T06:53:00Z

It seems that the memory is not enough for the ubuntu-latest

the test after test_horovod does not start for test_tune.py
the release test hang on when it comes to running ray_ddp_example.py with Tune

Does that mean for the tune, there is OOM issue?

JiahaoYao · 2022-08-18T22:09:48Z

[pytest on push/test_linux_ray_master_3]   ✅  Success - Install package
[pytest on push/test_linux_ray_master_3] ⭐  Run Test with Pytest
[pytest on push/test_linux_ray_master_3]   🐳  docker exec cmd=[bash --noprofile --norc -e -o pipefail /Users/jimmy/ScratchGym/Scratch/test0818/ray_lightning/workflow/4] user=
| /Users/jimmy/ScratchGym/Scratch/test0818/ray_lightning/ray_lightning/tests /Users/jimmy/ScratchGym/Scratch/test0818/ray_lightning
| ============================= test session starts ==============================
| platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.7.13/x64/bin/python
| cachedir: .pytest_cache
| rootdir: /Users/jimmy/ScratchGym/Scratch/test0818/ray_lightning
collected 7 items
|
| test_tune.py::test_tune_iteration_ddp PASSED                             [ 14%]
| test_tune.py::test_tune_iteration_horovod PASSED                         [ 28%]
| test_tune.py::test_checkpoint_ddp PASSED                                 [ 42%]
| test_tune.py::test_checkpoint_horovod PASSED                             [ 57%]
| test_tune.py::test_checkpoint_ddp_gpu SKIPPED (test requires multi-G...) [ 71%]
| test_tune.py::test_checkpoint_horovod_gpu SKIPPED (test requires mul...) [ 85%]
| test_tune.py::test_tune_iteration_ddp_gpu SKIPPED (test requires mul...) [100%]
|
| =============================== warnings summary ===============================
| ../../../../../../../../opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/horovod/torch/sync_batch_norm.py:33
| ../../../../../../../../opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/horovod/torch/sync_batch_norm.py:33
|   /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/horovod/torch/sync_batch_norm.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
|     LooseVersion(torch.__version__) >= LooseVersion('1.5.0') and
|
| ../../../../../../../../opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/horovod/torch/sync_batch_norm.py:34
| ../../../../../../../../opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/horovod/torch/sync_batch_norm.py:34
|   /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/horovod/torch/sync_batch_norm.py:34: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
|     LooseVersion(torch.__version__) <= LooseVersion('1.6.0')
|
| ../../../../../../../../opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/horovod/torch/sync_batch_norm.py:36
| ../../../../../../../../opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/horovod/torch/sync_batch_norm.py:36
|   /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/horovod/torch/sync_batch_norm.py:36: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
|     _SYNC_BN_V3 = LooseVersion(torch.__version__) >= LooseVersion('1.6.0')
|
| ../../../../../../../../opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/horovod/torch/sync_batch_norm.py:37
| ../../../../../../../../opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/horovod/torch/sync_batch_norm.py:37
|   /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/horovod/torch/sync_batch_norm.py:37: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
|     _SYNC_BN_V4 = LooseVersion(torch.__version__) >= LooseVersion('1.9.0')
|
| ../../../../../../../../opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/torch/utils/tensorboard/__init__.py:5
|   /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/torch/utils/tensorboard/__init__.py:5: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
|     tensorboard.__version__
|
| ../../../../../../../../opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/torch/utils/tensorboard/__init__.py:6
|   /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/torch/utils/tensorboard/__init__.py:6: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
|     ) < LooseVersion("1.15"):
|
| ray_lightning/tests/test_tune.py::test_tune_iteration_ddp
| ray_lightning/tests/test_tune.py::test_tune_iteration_ddp
| ray_lightning/tests/test_tune.py::test_tune_iteration_horovod
| ray_lightning/tests/test_tune.py::test_tune_iteration_horovod
| ray_lightning/tests/test_tune.py::test_checkpoint_ddp
| ray_lightning/tests/test_tune.py::test_checkpoint_horovod
|   /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/ray/util/placement_group.py:80: DeprecationWarning: placement_group parameter is deprecated. Use scheduling_strategy=PlacementGroupSchedulingStrategy(...) instead, see the usage at https://docs.ray.io/en/master/ray-core/package-ref.html#ray-remote.
|     ).remote(self)
|
| ray_lightning/tests/test_tune.py::test_tune_iteration_ddp
| ray_lightning/tests/test_tune.py::test_tune_iteration_horovod
| ray_lightning/tests/test_tune.py::test_checkpoint_ddp
| ray_lightning/tests/test_tune.py::test_checkpoint_horovod
|   /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/ray/_private/ray_option_utils.py:291: DeprecationWarning: Setting 'object_store_memory' for actors is deprecated since it doesn't actually reserve the required object store memory. Use object spilling that's enabled by default (https://docs.ray.io/en/master/ray-core/objects/object-spilling.html) instead to bypass the object store memory size limitation.
|     stacklevel=1,
|
| ray_lightning/tests/test_tune.py::test_tune_iteration_ddp
| ray_lightning/tests/test_tune.py::test_tune_iteration_horovod
| ray_lightning/tests/test_tune.py::test_checkpoint_ddp
| ray_lightning/tests/test_tune.py::test_checkpoint_horovod
|   /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/ray/actor.py:637: DeprecationWarning: placement_group parameter is deprecated. Use scheduling_strategy=PlacementGroupSchedulingStrategy(...) instead, see the usage at https://docs.ray.io/en/master/ray-core/package-ref.html#ray-remote.
|     return actor_cls._remote(args=args, kwargs=kwargs, **updated_options)
|
| ray_lightning/tests/test_tune.py::test_tune_iteration_ddp
| ray_lightning/tests/test_tune.py::test_tune_iteration_horovod
| ray_lightning/tests/test_tune.py::test_checkpoint_ddp
| ray_lightning/tests/test_tune.py::test_checkpoint_horovod
|   /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/ray/actor.py:637: DeprecationWarning: placement_group_bundle_index parameter is deprecated. Use scheduling_strategy=PlacementGroupSchedulingStrategy(...) instead, see the usage at https://docs.ray.io/en/master/ray-core/package-ref.html#ray-remote.
|     return actor_cls._remote(args=args, kwargs=kwargs, **updated_options)
|
| ray_lightning/tests/test_tune.py::test_tune_iteration_ddp
| ray_lightning/tests/test_tune.py::test_tune_iteration_horovod
| ray_lightning/tests/test_tune.py::test_checkpoint_ddp
| ray_lightning/tests/test_tune.py::test_checkpoint_horovod
|   /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/ray/actor.py:637: DeprecationWarning: placement_group_capture_child_tasks parameter is deprecated. Use scheduling_strategy=PlacementGroupSchedulingStrategy(...) instead, see the usage at https://docs.ray.io/en/master/ray-core/package-ref.html#ray-remote.
|     return actor_cls._remote(args=args, kwargs=kwargs, **updated_options)
|
| -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
| ============================== slowest durations ===============================
| 12.07s call     ray_lightning/tests/test_tune.py::test_tune_iteration_ddp
| 11.92s call     ray_lightning/tests/test_tune.py::test_tune_iteration_horovod
| 9.61s call     ray_lightning/tests/test_tune.py::test_checkpoint_ddp
| 7.93s call     ray_lightning/tests/test_tune.py::test_checkpoint_horovod
| 3.66s setup    ray_lightning/tests/test_tune.py::test_tune_iteration_ddp
| 3.55s setup    ray_lightning/tests/test_tune.py::test_checkpoint_ddp
| 3.01s teardown ray_lightning/tests/test_tune.py::test_tune_iteration_ddp
| 2.97s teardown ray_lightning/tests/test_tune.py::test_tune_iteration_horovod
| 2.77s teardown ray_lightning/tests/test_tune.py::test_checkpoint_ddp
| 2.64s teardown ray_lightning/tests/test_tune.py::test_checkpoint_horovod
| 2.56s setup    ray_lightning/tests/test_tune.py::test_tune_iteration_horovod
| 2.46s setup    ray_lightning/tests/test_tune.py::test_checkpoint_horovod
|
| (6 durations < 0.005s hidden.  Use -vv to show these durations.)
| ============= 4 passed, 3 skipped, 32 warnings in 68.14s (0:01:08) =============
[pytest on push/test_linux_ray_master_3]   ✅  Success - Test with Pytest

marcosrdac · 2022-09-27T21:26:30Z

I just tested this PR and it worked fine on my cluster (training on 12 GPUs).

JiahaoYao · 2022-09-29T06:28:59Z

Any ideas to fix this in the ci test?

Requested labels: ubuntu-latest
Job defined at: ray-project/ray_lightning/.github/workflows/test.yaml@refs/pull/196/merge
Waiting for a runner to pick up this job...

Is there a typo mentioned here (https://github.com/orgs/community/discussions/31587)?

JiahaoYao

ready to run the ci test

sxjscience · 2022-09-30T08:14:38Z

The CI error does not seem to be related to the PR:

        if not _JSONARGPARSE_SIGNATURES_AVAILABLE:
            raise ModuleNotFoundError(
>               f"{_JSONARGPARSE_SIGNATURES_AVAILABLE}. Try `pip install -U 'jsonargparse[signatures]'`."
            )
E           ModuleNotFoundError: Requirement 'jsonargparse[signatures]>=4.12.0' not met, DistributionNotFound: The 'docstring-parser>=0.15; extra == "signatures"' distribution was not found and is required by jsonargparse. Try `pip install -U 'jsonargparse[signatures]'`.

/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/pytorch_lightning/cli.py:73: ModuleNotFoundError
=============================== warnings summary ===============================

.github/workflows/test.yaml

sxjscience · 2022-10-03T16:12:31Z

@JiahaoYao Any plan to finish this PR?

…4/entrypoints-object-has-no-attribute-get-digital-ocean

JiahaoYao · 2022-10-03T17:50:08Z

the hanging issue still remains for the release test

== Status ==
Current time: 2022-10-03 17:49:05 (running for 00:14:54.03)
Memory usage on this node: 1.7/6.8 GiB
Using FIFO scheduling algorithm.
Resources requested: 2.0/2 CPUs, 0/0 GPUs, 0.0/3.56 GiB heap, 0.0/1.78 GiB objects
Result logdir: /home/runner/ray_results/tune_mnist
Number of trials: 1/1 (1 RUNNING)
+-------------------------+----------+-----------------+--------------+-----------+-----------+------------+
| Trial name              | status   | loc             |   batch_size |   layer_1 |   layer_2 |         lr |
|-------------------------+----------+-----------------+--------------+-----------+-----------+------------|
| train_mnist_97d14_00000 | RUNNING  | 10.1.0.228:5153 |           32 |        64 |        64 | 0.00670904 |
+-------------------------+----------+-----------------+--------------+-----------+-----------+------------+
== Status ==
Current time: 2022-10-03 17:49:10 (running for 00:14:59.03)
Memory usage on this node: 1.7/6.8 GiB
Using FIFO scheduling algorithm.
Resources requested: 2.0/2 CPUs, 0/0 GPUs, 0.0/3.56 GiB heap, 0.0/1.78 GiB objects
Result logdir: /home/runner/ray_results/tune_mnist
Number of trials: 1/1 (1 RUNNING)
+-------------------------+----------+-----------------+--------------+-----------+-----------+------------+
| Trial name              | status   | loc             |   batch_size |   layer_1 |   layer_2 |         lr |
|-------------------------+----------+-----------------+--------------+-----------+-----------+------------|
| train_mnist_97d14_00000 | RUNNING  | 10.1.0.228:5153 |           32 |        64 |        64 | 0.00670904 |
+-------------------------+----------+-----------------+--------------+-----------+-----------+------------+

aga-relation · 2022-10-28T09:23:05Z

Any updates on this? :)

enochkan

approval was a mistake. Sorry for any inconvenience caused.

JiahaoYao added 9 commits August 11, 2022 23:43

update lightning

d9e6ea7

change gpu to cuda

5992c73

accelerator is successfully removed

63bb622

remove the accelerator

61a3775

progressive bar

db2f6ca

adding the transform

282b993

change the max step to be -1

fad6be9

fix ci

868688d

checkpoint

612d08f

sxjscience reviewed Aug 13, 2022

View reviewed changes

setup.py Outdated Show resolved Hide resolved

JiahaoYao added 8 commits August 13, 2022 05:17

checkpoint_callback

302f047

adding the version

5ba8227

fix 'MNISTDataModule' object has no attribute 'train_transforms' issue

878c4b9

fix the issue

7eb7c83

nit

d063b41

remove progress bar

bb5f8ff

remove accelerator

966d819

adding remote back

b3bc93d

JiahaoYao added 2 commits August 15, 2022 23:42

update bolts

c3f5ce8

kit start

e6e3817

JiahaoYao added 4 commits August 18, 2022 20:34

split the testing

d15f24c

Merge remote-tracking branch 'upstream/main' into rlt_1.7_0811

8bfdb20

put it back

6bb1acd

fix the ci here

d868524

JiahaoYao added 2 commits August 18, 2022 22:42

test memory

f7ed645

update the cpu number to 6

1c63f48

JiahaoYao added 3 commits August 19, 2022 00:32

adding the pip list

55e514e

addikng this

06b84c9

adding the debug

c607f0d

JiahaoYao added 3 commits September 29, 2022 06:10

Merge remote-tracking branch 'upstream/main' into rlt_1.7_0811

645fed2

update the lightning version

81fb6a4

rerun the ci test

ecd9fac

JiahaoYao commented Sep 29, 2022

View reviewed changes

sxjscience reviewed Sep 30, 2022

View reviewed changes

.github/workflows/test.yaml Outdated Show resolved Hide resolved

JiahaoYao added 7 commits October 3, 2022 16:51

adding the siginature pack

98ea680

new line

fc630a6

import lib breaks due eto https://stackoverflow.com/questions/7392956…

b0d4cd1

…4/entrypoints-object-has-no-attribute-get-digital-ocean

fix the lint

665e8fd

switch

3868cff

import failure

a277600

nit

c08a8f5

JiahaoYao mentioned this pull request Oct 3, 2022

[experiment] update the ray lightning to 1.7 #222

Open

sxjscience mentioned this pull request Oct 7, 2022

Temporarily skip AutoMM HPO tests (lightning 1.7) autogluon/autogluon#2192

Merged

enochkan approved these changes Feb 14, 2023

View reviewed changes

enochkan reviewed Feb 14, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support pytorch lightning 1.7 #196

support pytorch lightning 1.7 #196

JiahaoYao commented Aug 11, 2022

sxjscience commented Aug 13, 2022

JiahaoYao commented Aug 15, 2022

JiahaoYao commented Aug 16, 2022

JiahaoYao commented Aug 18, 2022

marcosrdac commented Sep 27, 2022

JiahaoYao commented Sep 29, 2022 •

edited

Loading

JiahaoYao left a comment

sxjscience commented Sep 30, 2022

sxjscience commented Oct 3, 2022 •

edited

Loading

JiahaoYao commented Oct 3, 2022

aga-relation commented Oct 28, 2022

enochkan left a comment

support pytorch lightning 1.7 #196

Are you sure you want to change the base?

support pytorch lightning 1.7 #196

Conversation

JiahaoYao commented Aug 11, 2022

sxjscience commented Aug 13, 2022

JiahaoYao commented Aug 15, 2022

JiahaoYao commented Aug 16, 2022

JiahaoYao commented Aug 18, 2022

marcosrdac commented Sep 27, 2022

JiahaoYao commented Sep 29, 2022 • edited Loading

JiahaoYao left a comment

Choose a reason for hiding this comment

sxjscience commented Sep 30, 2022

sxjscience commented Oct 3, 2022 • edited Loading

JiahaoYao commented Oct 3, 2022

aga-relation commented Oct 28, 2022

enochkan left a comment

Choose a reason for hiding this comment

JiahaoYao commented Sep 29, 2022 •

edited

Loading

sxjscience commented Oct 3, 2022 •

edited

Loading