Skip to content

Commit 5b5a5cc

Browse files
BordaBorda
authored
support python 3.9 (#4944)
* support python 3.9 * update CI * onnxruntime * . * . * onnxruntime * t 55 * t 75 * add script * use * onnx * onnx * onnx * whl * np * find * 21 * Apply suggestions from code review * Apply suggestions from code review * onnx * CI * req * ~ dockers * min * . * drop horovod * drop horovod * drop horovod * fix * fix * .
1 parent cca0eca commit 5b5a5cc

12 files changed

+88
-19
lines changed

‎.github/workflows/ci_dockers.yml

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -75,8 +75,8 @@ jobs:
7575
pytorch_version: 1.6
7676
- python_version: 3.8
7777
pytorch_version: 1.7
78-
#- python_version: 3.9
79-
# pytorch_version: 1.8
78+
# - python_version: 3.9
79+
# pytorch_version: 1.7
8080
steps:
8181
- name: Checkout
8282
uses: actions/checkout@v2
@@ -109,8 +109,10 @@ jobs:
109109
pytorch_version: 1.4
110110
- python_version: 3.7
111111
pytorch_version: 1.7
112-
- python_version: 3.7
112+
- python_version: 3.8
113113
pytorch_version: 1.8
114+
# - python_version: 3.9
115+
# pytorch_version: 1.8
114116
steps:
115117
- name: Checkout
116118
uses: actions/checkout@v2

‎.github/workflows/ci_pkg-install.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ jobs:
1717
matrix:
1818
# PyTorch 1.5 is failing on Win and bolts requires torchvision>=0.5
1919
os: [ubuntu-20.04, macOS-10.15 , windows-2019] #
20-
python-version: [3.6, 3.8]
20+
python-version: [3.6, 3.9]
2121

2222
steps:
2323
- uses: actions/checkout@v2

‎.github/workflows/ci_test-base.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ jobs:
1616
# max-parallel: 6
1717
matrix:
1818
os: [ubuntu-20.04, windows-2019, macOS-10.15]
19-
python-version: [3.7]
19+
python-version: [3.8]
2020

2121
# Timeout: https://stackoverflow.com/a/59076067/4521646
2222
timeout-minutes: 20

‎.github/workflows/ci_test-full.yml

Lines changed: 30 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,19 +15,24 @@ jobs:
1515
fail-fast: false
1616
matrix:
1717
os: [ubuntu-18.04, windows-2019, macOS-10.15]
18-
python-version: [3.6, 3.7, 3.8]
18+
python-version: [3.6, 3.7, 3.8, 3.9]
1919
requires: ['minimal', 'latest']
20+
exclude:
21+
- python-version: 3.9
22+
requires: 'minimal'
2023

2124
# Timeout: https://stackoverflow.com/a/59076067/4521646
22-
timeout-minutes: 35 # TODO: the macOS is taking too long, probably caching did not work...
25+
# TODO: the macOS is taking too long, probably caching did not work...
26+
timeout-minutes: 40
27+
2328
steps:
2429
- uses: actions/checkout@v2
2530
- name: Set up Python ${{ matrix.python-version }}
2631
uses: actions/setup-python@v2
2732
with:
2833
python-version: ${{ matrix.python-version }}
2934

30-
- name: Update Pip
35+
- name: Update pip
3136
run: |
3237
# todo: unfreeze PIP after resolving minimal dependencies
3338
pip install --quiet "pip==20.1" --upgrade --user # needed for get pip cacher folder
@@ -48,6 +53,19 @@ jobs:
4853
open(fname, 'w').writelines(lines)
4954
shell: python
5055

56+
# todo: re-enable when allow testing py 3.9 with min config, atm some Hydra issues
57+
#- name: Adjust minimal for Python 3.9
58+
# if: matrix.requires == 'minimal' && matrix.python-version == 3.9
59+
# run: |
60+
# import re
61+
# def _req(fname, ptn, ver):
62+
# req = re.sub(ptn, ver, open(fname).read())
63+
# open(fname, 'w').write(req)
64+
#
65+
# _req('requirements.txt', r'torch>=[\d\.]+', 'torch>=1.8.0')
66+
# _req('requirements/extra.txt', r'onnxruntime>=[\d\.]+', 'onnxruntime>=1.7.0')
67+
# shell: python
68+
5169
- name: Set min. dependencies
5270
if: matrix.requires == 'minimal'
5371
run: |
@@ -84,7 +102,6 @@ jobs:
84102
${{ runner.os }}-pip-py${{ matrix.python-version }}-${{ matrix.requires }}-
85103
86104
- name: Pull checkpoints from S3
87-
# todo: consider adding some caching, but ATM all models have less then 100KB
88105
run: |
89106
cd legacy
90107
# wget is simpler but does not work on Windows
@@ -93,6 +110,13 @@ jobs:
93110
unzip -o checkpoints.zip
94111
ls -l checkpoints/
95112
113+
# todo: re-enable testing with Horovod
114+
- name: py3.9 - temp skip Horovod
115+
if: matrix.python-version == 3.9
116+
run: |
117+
# pip uninstall -y horovod
118+
python -c "fname = 'requirements/extra.txt' ; lines = [line for line in open(fname).readlines() if not line.startswith('horovod')] ; open(fname, 'w').writelines(lines)"
119+
96120
- name: Install dependencies
97121
env:
98122
# MAKEFLAGS: "-j2"
@@ -112,7 +136,8 @@ jobs:
112136
shell: bash
113137

114138
- name: Reinstall Horovod if necessary
115-
if: runner.os != 'windows'
139+
# todo: re-enable horovod on py3.9 when it will be supported
140+
if: runner.os != 'windows' && matrix.python-version != 3.9
116141
env:
117142
HOROVOD_BUILD_ARCH_FLAGS: "-mfma"
118143
run: |

‎.github/workflows/release-docker.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ jobs:
1414
fail-fast: false
1515
matrix:
1616
python_version: [3.6, 3.7, 3.8]
17-
pytorch_version: [1.4, 1.5, 1.6, 1.7]
17+
pytorch_version: [1.4, 1.5, 1.6, 1.7, 1.8]
1818
steps:
1919
- name: Checkout
2020
uses: actions/checkout@v2

‎README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -79,9 +79,9 @@ Lightning is rigurously tested across multiple GPUs, TPUs CPUs and against major
7979
| Conda py3.7 [linux] | [![PyTorch & Conda](https://github.com/PyTorchLightning/pytorch-lightning/workflows/PyTorch%20&%20Conda/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22PyTorch+%26+Conda%22+branch%3Amaster) | [![PyTorch & Conda](https://github.com/PyTorchLightning/pytorch-lightning/workflows/PyTorch%20&%20Conda/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22PyTorch+%26+Conda%22+branch%3Amaster) | [![PyTorch & Conda](https://github.com/PyTorchLightning/pytorch-lightning/workflows/PyTorch%20&%20Conda/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22PyTorch+%26+Conda%22+branch%3Amaster) | [![PyTorch & Conda](https://github.com/PyTorchLightning/pytorch-lightning/workflows/PyTorch%20&%20Conda/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22PyTorch+%26+Conda%22+branch%3Amaster) | [![PyTorch & Conda](https://github.com/PyTorchLightning/pytorch-lightning/workflows/PyTorch%20&%20Conda/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22PyTorch+%26+Conda%22+branch%3Amaster) |
8080
| Linux py3.7 [GPUs**] | - | - | [![Build Status](https://dev.azure.com/PytorchLightning/pytorch-lightning/_apis/build/status/PyTorchLightning.pytorch-lightning?branchName=master)](https://dev.azure.com/PytorchLightning/pytorch-lightning/_build/latest?definitionId=2&branchName=master) | - | - |
8181
| Linux py3.{6,7} [TPUs***] | - | - | [![TPU tests](https://github.com/PyTorchLightning/pytorch-lightning/workflows/TPU%20tests/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22TPU+tests%22+branch%3Amaster) | [![TPU tests](https://github.com/PyTorchLightning/pytorch-lightning/workflows/TPU%20tests/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22TPU+tests%22+branch%3Amaster) |
82-
| Linux py3.{6,7} | [![CI complete testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20complete%20testing/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - | - | [![CI complete testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20complete%20testing/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - |
83-
| OSX py3.{6,7,8} | - | [![CI complete testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20complete%20testing/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - | [![CI complete testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20complete%20testing/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - |
84-
| Windows py3.{6,7,8} | [![CI complete testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20complete%20testing/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - | - | [![CI complete testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20complete%20testing/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - |
82+
| Linux py3.{6,7,8,9} | [![CI complete testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20complete%20testing/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - | - | [![CI complete testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20complete%20testing/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - |
83+
| OSX py3.{6,7,8,9} | - | [![CI complete testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20complete%20testing/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - | [![CI complete testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20complete%20testing/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - |
84+
| Windows py3.{6,7,8,9} | [![CI complete testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20complete%20testing/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - | - | [![CI complete testing](https://github.com/PyTorchLightning/pytorch-lightning/workflows/CI%20complete%20testing/badge.svg?branch=master&event=push)](https://github.com/PyTorchLightning/pytorch-lightning/actions?query=workflow%3A%22CI+testing%22) | - |
8585

8686
- _\** tests run on two NVIDIA K80_
8787
- _\*** tests run on Google GKE TPUv2/3_

‎requirements/install_AMP.sh renamed to ‎requirements/install_Apex.sh

File renamed without changes.

‎requirements/install_ONNX.sh

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#!/usr/bin/env bash
2+
3+
ROOT=$PWD
4+
5+
# python -m pip install protobuf
6+
# git clone --recursive https://github.com/onnx/onnx.git
7+
# cd onnx
8+
# python setup.py bdist_wheel
9+
# pip install --upgrade dist/*.whl
10+
# cd $ROOT
11+
# rm -rf onnx
12+
13+
14+
# https://github.com/microsoft/onnxruntime/blob/master/BUILD.md
15+
git clone --recursive https://github.com/Microsoft/onnxruntime
16+
cd onnxruntime
17+
export ONNX_ML=1
18+
pip install setuptools wheel numpy
19+
20+
if [[ "$OSTYPE" == "linux-gnu"* ]]; then
21+
./build.sh --config RelWithDebInfo --build_shared_lib --build_wheel --parallel
22+
elif [[ "$OSTYPE" == "darwin"* ]]; then
23+
# Mac OSX
24+
./build.sh --config RelWithDebInfo --build_shared_lib --build_wheel --parallel --use_xcode
25+
elif [[ "$OSTYPE" == "cygwin" ]]; then
26+
# POSIX compatibility layer and Linux environment emulation for Windows
27+
./build.sh --config RelWithDebInfo --build_shared_lib --build_wheel --parallel
28+
elif [[ "$OSTYPE" == "msys" ]]; then
29+
# Lightweight shell and GNU utilities compiled for Windows (part of MinGW)
30+
.\build.bat --config RelWithDebInfo --build_shared_lib --build_wheel --parallel
31+
elif [[ "$OSTYPE" == "win32" ]]; then
32+
.\build.bat --config RelWithDebInfo --build_shared_lib --build_wheel --parallel
33+
else
34+
echo $OSTYPE # Unknown.
35+
fi
36+
37+
find . -name "*.whl"
38+
pip install --upgrade $(find . -name "*.whl")
39+
40+
cd $ROOT
41+
rm -rf onnxruntime

‎requirements/test.txt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,12 +6,12 @@ pytest>=6.0
66
flake8>=3.6
77
check-manifest
88
twine==3.2
9-
scikit-learn>=0.22.2
10-
scikit-image>=0.17.2
119
isort>=5.6.4
1210
mypy>=0.720, <0.800
1311
pre-commit>=1.0
1412

1513
cloudpickle>=1.3
14+
scikit-learn>0.22.1
15+
scikit-image>0.17.1
1616
nltk>=3.3
1717
pandas # needed in benchmarks

‎setup.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,5 +110,6 @@
110110
'Programming Language :: Python :: 3.6',
111111
'Programming Language :: Python :: 3.7',
112112
'Programming Language :: Python :: 3.8',
113+
'Programming Language :: Python :: 3.9',
113114
],
114115
)

‎tests/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ git clone https://github.com/PyTorchLightning/pytorch-lightning
1717
cd pytorch-lightning
1818

1919
# install AMP support
20-
bash requirements/install_AMP.sh
20+
bash requirements/install_Apex.sh
2121

2222
# install dev deps
2323
pip install -r requirements/devel.txt

‎tests/models/test_horovod.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ def _run_horovod(trainer_options, on_gpu=False):
6363
assert exit_code == 0
6464

6565

66-
@RunIf(skip_windows=True)
66+
@RunIf(skip_windows=True, horovod=True)
6767
def test_horovod_cpu(tmpdir):
6868
"""Test Horovod running multi-process on CPU."""
6969
trainer_options = dict(
@@ -80,7 +80,7 @@ def test_horovod_cpu(tmpdir):
8080
_run_horovod(trainer_options)
8181

8282

83-
@RunIf(skip_windows=True)
83+
@RunIf(skip_windows=True, horovod=True)
8484
def test_horovod_cpu_implicit(tmpdir):
8585
"""Test Horovod without specifying a backend, inferring from env set by `horovodrun`."""
8686
trainer_options = dict(

0 commit comments

Comments
 (0)