Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix code inspection notebook #832

Merged
merged 6 commits into from
Feb 5, 2024
Merged
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 48 additions & 2 deletions src/fondant/pipeline/lightweight_component.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@

logger = logging.getLogger(__name__)


MIN_PYTHON_VERSION = (3, 8)
MAX_PYTHON_VERSION = (3, 11)

Expand Down Expand Up @@ -264,6 +263,39 @@ def consumes(cls) -> t.Optional[t.Dict[str, t.Dict[t.Any, t.Any]]]:
return wrapper


def new_getfile(_object, _old_getfile=inspect.getfile):
if not inspect.isclass(_object):
return _old_getfile(_object)

# Lookup by parent module (as in current inspect)
if hasattr(_object, "__module__"):
object_ = sys.modules.get(_object.__module__)
if hasattr(object_, "__file__"):
return object_.__file__

# If parent module is __main__, lookup by methods (NEW)
for name, member in inspect.getmembers(_object):
if (
inspect.isfunction(member)
and _object.__qualname__ + "." + member.__name__ == member.__qualname__
):
return inspect.getfile(member)

msg = f"Source for {_object!r} not found"
raise TypeError(msg)


def is_running_in_notebook():
"""Check if the code is running in a Jupyter notebook."""
try:
from IPython import get_ipython

shell = get_ipython().__class__.__name__
return shell == "ZMQInteractiveShell"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about running in an IPython terminal instead of a notebook? (if shell == "TerminalInteractiveShell")

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this fix work for the other shells as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested on my IDE python consol shell (PyDevTerminalInteractiveShell), jupter notebook (ZMQInteractiveShell) and colab (Shell). I presume those are the main 3 options that will be used for compilation (even IDE console seems a bit unlikely)

except ModuleNotFoundError:
return False


def build_python_script(component_cls: t.Type[Component]) -> str:
"""Build a self-contained python script for the provided component class, which will act as
the `src/main.py` script to execute the component.
Expand All @@ -281,7 +313,21 @@ def build_python_script(component_cls: t.Type[Component]) -> str:
""",
)

component_source = inspect.getsource(component_cls)
if is_running_in_notebook():
from IPython.core.magics.code import extract_symbols

inspect.getfile = new_getfile
component_source = "".join(
inspect.linecache.getlines( # type: ignore[attr-defined]
new_getfile(component_cls),
),
)
component_source = extract_symbols(component_source, component_cls.__name__)[0][
0
]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Y u do this Black 😢

Suggested change
component_source = extract_symbols(component_source, component_cls.__name__)[0][
0
]
component_source = extract_symbols(
component_source,
component_cls.__name__,
)[0][0]

Can you try changing it to this and see if Black keeps it this way? 😛

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

haha it does this

  component_source = extract_symbols(
      component_source,
      component_cls.__name__,
  )[
      0
  ][0]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'll break it into two lines

else:
component_source = inspect.getsource(component_cls)

component_source = textwrap.dedent(component_source)
component_source_lines = component_source.split("\n")

Expand Down
Loading