Add auto-generated pyi files to Metaflow (#1557)

* Fix some docstrings and add Add To Current

* Docstring changes

* More docstrings

* Stub generator script

* More docstring fixes

* Improve stub generator script

Now:
  - deals with static and class methods
  - better handling of forward types or types and files with __annotations__
  - better handling of default values

* Add specialized function in the client to get proper return types

* Fix invalid docstrings

* Fixup more stuff in stub generator

* make docstrings consistent

* Small fixes

* Move current.py to metaflow_current.py

* Minor tweaks

* Added MANIFEST.in

* Fix issues with numpydoc compilation

* Updated stub generator to include step and type check decorators better

* Addressed comments

* Added a stub command; moved script to new location. (#1682)

* Added a stub command; moved script to new location.

* validation of sample flow with mypy + packaging (#1676)

* validate sample flow with mypy

* also install metaflow in CI

* change commands

* package generated stubs with metaflow

* don't do editable install

* re-introduce setup.py

* update publish action

* point to dist

* check common cases with pytest-mypy plugin

* Updated stub tests; added kw only args to stub generator

* remove test flow

* kubernetes validity

* add tests for environment, card, catch, conda

* add tests for timeout, secrets, retry, resources decorators

* add tests for conda_base, schedule, trigger, trigger_on_finish decorators

* Get tests to pass again

---------

Co-authored-by: Romain Cledat <[email protected]>

* fix checking of stubs package (#1689)

* Properly uninstall packages that provide stubs before installing a new one.

* Add a few more cases to the stub generator

* add tests for pypi and pypi_base decorators (#1694)

* Remove print; better setup.py

---------

Co-authored-by: madhur-ob <[email protected]>

* Small fixes to be better with PyCharm

* Example test with more flexible verification for mypy

* use regex for mypy tests (#1697)

* use regex for mypy tests

* fix regex

* edit testing CI (#1699)

* separate workflow for testing stubs (#1701)

* separate workflow for testing stubs

* remove 3.5 and 3.6

* fix test stubs workflow (#1702)

* install build via pip

* install pytest

* add timeout (#1703)

* remove on_retry_command

* Fix issue with stub generator for TypeVar

* Add extra requires to main setup.py

* publish stubs package before (#1705)

* Change color of 'valid'

---------

Co-authored-by: Ville Tuulos <[email protected]>
Co-authored-by: madhur-ob <[email protected]>
Co-authored-by: Madhur Tandon <[email protected]>

.github/workflows/publish.yml

@@ -18,12 +18,24 @@ jobs:
       run: |
         python3 -m pip install --upgrade pip
         pip3 install setuptools wheel twine
-    - name: Build package
+    - name: Install metaflow
+      run: pip install .
+    - name: Generate Stubs
+      run: cd ./stubs && rm -rf metaflow-stubs/ && python -c "from metaflow.cmd.develop.stub_generator import StubGenerator; StubGenerator('./metaflow-stubs').write_out()" && cd -
+    - name: Build metaflow package
       run: |
         python3 setup.py sdist bdist_wheel --universal
-    - name: Publish package
+    - name: Build metaflow-stubs package
+      run: |
+        cd ./stubs && python3 setup.py sdist bdist_wheel --universal && cd -
+    - name: Publish metaflow-stubs package
+      uses: pypa/gh-action-pypi-publish@9b8e7336db3f96a2939a3e9fa827c62f466ca60d # master
+      with:
+        user: __token__
+        password: ${{ secrets.pypi_password }}
+        packages-dir: ./stubs/dist
+    - name: Publish metaflow package
       uses: pypa/gh-action-pypi-publish@9b8e7336db3f96a2939a3e9fa827c62f466ca60d # master
       with:
         user: __token__
         password: ${{ secrets.pypi_password }}
-

.github/workflows/test-stubs.yml

@@ -0,0 +1,48 @@
+name: Test Stubs
+on:
+  push:
+    branches:
+    - master
+  pull_request:
+    branches:
+    - master
+  workflow_call:
+
+permissions: read-all
+
+jobs:
+  Python:
+    name: core / Python ${{ matrix.ver }} on ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-20.04, macos-latest]
+        ver: ['3.7', '3.8', '3.9', '3.10', '3.11', '3.12']
+
+    steps:
+    - uses: actions/checkout@ee0669bd1cc54295c223e0bb666b733df41de1c5 # v2.7.0
+
+    - name: Set up Python
+      uses: actions/setup-python@75f3110429a8c05be0e1bf360334e4cced2b63fa # v2.3.3
+      with:
+        python-version: ${{ matrix.ver }}
+
+    - name: Install Python ${{ matrix.ver }} dependencies
+      run: |
+        python3 -m pip install --upgrade pip setuptools
+        python3 -m pip install pytest build mypy pytest-mypy-plugins
+
+    - name: Install metaflow
+      run: pip install .
+
+    - name: Install metaflow-stubs
+      run: metaflow develop stubs install --force
+
+    - name: Run mypy tests
+      uses: nick-fields/retry@v2
+      with:
+        max_attempts: 2
+        timeout_minutes: 3
+        retry_on: error
+        command: cd ./stubs && pytest --mypy-ini-file test/setup.cfg --mypy-only-local-stub && cd -

metaflow/__init__.py

@@ -98,7 +98,7 @@ class and related decorators.
 from .metaflow_profile import profile
 
 # current runtime singleton
-from .current import current
+from .metaflow_current import current
 
 # Flow spec
 from .flowspec import FlowSpec

metaflow/cli.py

@@ -13,7 +13,7 @@
 from . import decorators
 from . import metaflow_version
 from . import namespace
-from . import current
+from .metaflow_current import current
 from .cli_args import cli_args
 from .tagging_util import validate_tags
 from .util import (
@@ -235,7 +235,6 @@ def output_dot(obj):
 )
 @click.pass_obj
 def dump(obj, input_path, private=None, max_value_size=None, include=None, file=None):
-
     output = {}
     kwargs = {
         "show_private": private,
@@ -738,7 +737,6 @@ def resume(
     decospecs=None,
     run_id_file=None,
 ):
-
     before_run(obj, tags, decospecs + obj.environment.decospecs())
 
     if origin_run_id is None:
@@ -824,7 +822,6 @@ def run(
     user_namespace=None,
     **kwargs
 ):
-
     if user_namespace is not None:
         namespace(user_namespace or None)
     before_run(obj, tags, decospecs + obj.environment.decospecs())