Add auto-generated pyi files to Metaflow #1557

romain-intel · 2023-09-28T19:48:11Z

No description provided.

romain-intel

Added a few comments to make reviewing this easier.

romain-intel · 2023-10-25T06:19:39Z

metaflow/current.py

@@ -1,14 +1,13 @@
 from collections import namedtuple
 import os
-from typing import Any, Optional
+from typing import Any, Optional, TYPE_CHECKING


note: for full disclosure, TYPE_CHECKING was introduced in 3.5.2. I feel it should still be OK and we can still support 3.5 (just 3.5.2+)

romain-intel · 2023-10-25T06:20:11Z

metaflow/client/core.py

@@ -203,6 +213,9 @@ def default_namespace() -> str:
    return get_namespace()


+MetaflowArtifacts = NamedTuple


This is just to give it a proper type so that there are no issues in the generated pyi file. It's not very informative but I don't think we can do much better.

romain-intel · 2023-10-25T06:20:35Z

metaflow/client/core.py

@@ -981,6 +950,50 @@ def __setstate__(self, state):
        super(DataArtifact, self).__setstate__(state)


+class MetaflowData(object):


Moved below to allow types to find each other.

romain-intel · 2023-10-25T06:20:54Z

metaflow/client/core.py

+        self,
+        stream: str,
+        as_unicode: bool = True,
+        meta_dict: Optional[Dict[str, Any]] = None,


Fixed adding optional

romain-intel · 2023-10-25T06:21:29Z

metaflow/client/core.py

@@ -337,17 +350,17 @@ def _get_object(self, *path_components):
            raise MetaflowNotFound("%s does not exist" % self)
        return result

-    def __iter__(self) -> Iterable["MetaflowObject"]:
+    def __iter__(self) -> Iterator["MetaflowObject"]:


I changed all things that have yield to Yields in the docstring and also to return an Iterator (proper return type for yields).

romain-intel · 2023-10-25T06:28:50Z

metaflow/plugins/datatools/s3/s3.py

@@ -143,7 +143,7 @@ def __init__(
        content_type: Optional[str] = None,
        metadata: Optional[Dict[str, str]] = None,
        range_info: Optional[RangeInfo] = None,
-        last_modified: int = None,
+        last_modified: Optional[int] = None,


added proper type.

romain-intel · 2023-10-25T06:30:00Z

metaflow/plugins/events_decorator.py

+
+    MF Add To Current
+    -----------------
+    trigger -> metaflow.events.Trigger


notice that both decorators can add the same thing. The definitions should be the same (one is just picked -- no checks). It will say that this applies to BOTH decorators.

romain-intel · 2023-10-25T06:30:10Z

metaflow/plugins/events_decorator.py

+
+        Returns
+        -------
+            Trigger


TODO: unindent.

romain-intel · 2023-10-25T06:30:47Z

metaflow/plugins/project_decorator.py

+
+    MF Add To Current
+    -----------------
+    project_name -> str


add multiple properties. Blank line between properties is optional I think but makes things clearer.

romain-intel · 2023-10-25T06:31:38Z

stubs/setup.py

@@ -0,0 +1,46 @@
+from setuptools import find_packages, setup
+
+version = "2.10.2"


we should probably read this from common location.

Now: - deals with static and class methods - better handling of forward types or types and files with __annotations__ - better handling of default values

nflx-mf-bot · 2024-01-18T21:28:12Z

Testing[602] @ 7b279fa

savingoyal

lgtm - few minor comments

savingoyal · 2024-01-23T16:03:50Z

metaflow/client/core.py

+        Yields
+        ------
+        Task
+            Control Task object for this step


nit - objects

savingoyal · 2024-01-23T16:09:14Z

metaflow/plugins/project_decorator.py

@@ -25,6 +25,50 @@ class ProjectDecorator(FlowDecorator):
        Project name. Make sure that the name is unique amongst all
        projects that use the same production scheduler. The name may
        contain only lowercase alphanumeric characters and underscores.
+
+    MF Add To Current


MF Add To Current - is there a special significance to this string?

Yes, this is used to define a section to add fields to the current object. It's non standard and just our thing.

savingoyal · 2024-01-23T16:09:59Z

stubs/setup.py

+    include_package_data=True,
+    name="metaflow-stubs",
+    version=version,
+    description="Metaflow: More Data Science, Less Engineering",


Change the description to notate that this is a stubs package

savingoyal · 2024-01-23T16:10:19Z

stubs/setup.py

+    name="metaflow-stubs",
+    version=version,
+    description="Metaflow: More Data Science, Less Engineering",
+    long_description=open("../README.md").read(),


This long description does not apply to this package

savingoyal · 2024-01-23T16:10:28Z

stubs/setup.py

+        "Programming Language :: Python :: 3.8",
+        "Programming Language :: Python :: 3.9",
+        "Programming Language :: Python :: 3.10",
+        "Programming Language :: Python :: 3.11",


savingoyal · 2024-01-23T16:12:18Z

metaflow/decorators.py

-def step(f):
+FlowSpecDerived = TypeVar("FlowSpecDerived", bound=FlowSpec)
+
+StepFlag = NewType("StepFlag", bool)


Can you add some comments here around the necessity for StepFlag?

* Added a stub command; moved script to new location. * validation of sample flow with mypy + packaging (#1676) * validate sample flow with mypy * also install metaflow in CI * change commands * package generated stubs with metaflow * don't do editable install * re-introduce setup.py * update publish action * point to dist * check common cases with pytest-mypy plugin * Updated stub tests; added kw only args to stub generator * remove test flow * kubernetes validity * add tests for environment, card, catch, conda * add tests for timeout, secrets, retry, resources decorators * add tests for conda_base, schedule, trigger, trigger_on_finish decorators * Get tests to pass again --------- Co-authored-by: Romain Cledat <[email protected]> * fix checking of stubs package (#1689) * Properly uninstall packages that provide stubs before installing a new one. * Add a few more cases to the stub generator * add tests for pypi and pypi_base decorators (#1694) * Remove print; better setup.py --------- Co-authored-by: madhur-ob <[email protected]>

* use regex for mypy tests * fix regex

* separate workflow for testing stubs * remove 3.5 and 3.6

* install build via pip * install pytest

romain-intel requested a review from tuulos September 28, 2023 19:48

romain-intel force-pushed the feat/autodocs branch from 730671a to 59ca48c Compare October 3, 2023 16:01

romain-intel force-pushed the feat/autodocs branch from 59ca48c to 7a41cec Compare October 17, 2023 07:51

romain-intel commented Oct 25, 2023

View reviewed changes

romain-intel mentioned this pull request Oct 26, 2023

feat: Allow the use of *-stubs packages mkdocstrings/griffe#221

Merged

savingoyal requested a review from saikonen October 30, 2023 15:51

romain-intel force-pushed the feat/autodocs branch from cbefb45 to 48ca1fa Compare November 3, 2023 05:53

romain-intel changed the title ~~[WIP] Working on adding auto-generated pyi files to Metaflow~~ Add auto-generated pyi files to Metaflow Nov 3, 2023

romain-intel and others added 11 commits December 15, 2023 19:57

Fix some docstrings and add Add To Current

0e65315

Docstring changes

9bda196

More docstrings

cfc12a6

Stub generator script

d7f17ca

More docstring fixes

fae3825

Improve stub generator script

3c589ad

Now: - deals with static and class methods - better handling of forward types or types and files with __annotations__ - better handling of default values

Add specialized function in the client to get proper return types

d543639

Fix invalid docstrings

c743b9f

Fixup more stuff in stub generator

43e07e7

make docstrings consistent

cd1bb4e

Small fixes

211d6e2

romain-intel force-pushed the feat/autodocs branch from aa3c5ec to 211d6e2 Compare January 4, 2024 08:34

romain-intel added 3 commits January 5, 2024 10:59

Move current.py to metaflow_current.py

7aae809

Minor tweaks

469d02a

Added MANIFEST.in

f67d4d9

romain-intel mentioned this pull request Jan 11, 2024

IDEs (e.g. PyCharm) don't recognize Metaflow decorators #354

Closed

romain-intel added 2 commits January 12, 2024 00:46

Fix issues with numpydoc compilation

772fcda

Updated stub generator to include step and type check decorators better

7b279fa

romain-intel added the testable label Jan 18, 2024

nflx-mf-bot removed the testable label Jan 18, 2024

romain-intel added the testable label Jan 18, 2024

nflx-mf-bot removed the testable label Jan 18, 2024

savingoyal previously approved these changes Jan 23, 2024

View reviewed changes

Addressed comments

98140e5

romain-intel dismissed savingoyal’s stale review via 98140e5 January 23, 2024 19:39

romain-intel and others added 14 commits January 23, 2024 15:20

Merge remote-tracking branch 'origin/master' into feat/autodocs

6ebcd3c

Small fixes to be better with PyCharm

19a30b0

Example test with more flexible verification for mypy

eda6382

use regex for mypy tests (#1697)

0da1858

* use regex for mypy tests * fix regex

edit testing CI (#1699)

a40e971

separate workflow for testing stubs (#1701)

e9ddfc9

* separate workflow for testing stubs * remove 3.5 and 3.6

fix test stubs workflow (#1702)

94a936e

* install build via pip * install pytest

add timeout (#1703)

a694136

remove on_retry_command

1945812

Fix issue with stub generator for TypeVar

d21c8ce

Add extra requires to main setup.py

54766e9

publish stubs package before (#1705)

a00c663

Change color of 'valid'

f8e05cd

romain-intel merged commit 0bb543d into master Jan 29, 2024
32 of 34 checks passed

romain-intel deleted the feat/autodocs branch January 29, 2024 23:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add auto-generated pyi files to Metaflow #1557

Add auto-generated pyi files to Metaflow #1557

romain-intel commented Sep 28, 2023

romain-intel left a comment

romain-intel Oct 25, 2023

romain-intel Oct 25, 2023

romain-intel Oct 25, 2023

romain-intel Oct 25, 2023

romain-intel Oct 25, 2023

romain-intel Oct 25, 2023

romain-intel Oct 25, 2023

romain-intel Oct 25, 2023

romain-intel Oct 25, 2023

romain-intel Oct 25, 2023

nflx-mf-bot commented Jan 18, 2024

savingoyal left a comment

savingoyal Jan 23, 2024

savingoyal Jan 23, 2024

romain-intel Jan 23, 2024

savingoyal Jan 23, 2024

savingoyal Jan 23, 2024

savingoyal Jan 23, 2024

savingoyal Jan 23, 2024

		@@ -203,6 +213,9 @@ def default_namespace() -> str:
		return get_namespace()


		MetaflowArtifacts = NamedTuple

		@@ -981,6 +950,50 @@ def __setstate__(self, state):
		super(DataArtifact, self).__setstate__(state)


		class MetaflowData(object):

		@@ -0,0 +1,46 @@
		from setuptools import find_packages, setup

		version = "2.10.2"

Add auto-generated pyi files to Metaflow #1557

Add auto-generated pyi files to Metaflow #1557

Conversation

romain-intel commented Sep 28, 2023

romain-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nflx-mf-bot commented Jan 18, 2024

savingoyal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment