Skip to content

Commit

Permalink
Fix code inspection notebook (#832)
Browse files Browse the repository at this point in the history
When running the lightweight components in a notebook, the component
class code inspection fails with error `source code is not available`.
Full error log here
[log.txt](https://github.com/ml6team/fondant/files/14114022/new.3.txt)

The error is related to the fact that `inspect` relies on the class
located in a source file which is not the case in a notebook where
everything runs interactively. This is described in details
[here](https://stackoverflow.com/questions/35854373/python-get-source-code-of-class-using-inspect#:~:text=getsource%20works%20by%20actually%20opening%20the%20source%20file%20that%20defines%20the%20object%20you%27re%20looking%20at.)

Implemented the workaround suggested
[here](ipython/ipython#11249)

---------

Co-authored-by: Robbe Sneyders <[email protected]>
  • Loading branch information
PhilippeMoussalli and RobbeSneyders authored Feb 5, 2024
1 parent 31ce723 commit 48cd7e0
Showing 1 changed file with 49 additions and 2 deletions.
51 changes: 49 additions & 2 deletions src/fondant/pipeline/lightweight_component.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@

logger = logging.getLogger(__name__)


MIN_PYTHON_VERSION = (3, 8)
MAX_PYTHON_VERSION = (3, 11)

Expand Down Expand Up @@ -264,6 +263,38 @@ def consumes(cls) -> t.Optional[t.Dict[str, t.Dict[t.Any, t.Any]]]:
return wrapper


def new_getfile(_object, _old_getfile=inspect.getfile):
if not inspect.isclass(_object):
return _old_getfile(_object)

# Lookup by parent module (as in current inspect)
if hasattr(_object, "__module__"):
object_ = sys.modules.get(_object.__module__)
if hasattr(object_, "__file__"):
return object_.__file__

# If parent module is __main__, lookup by methods (NEW)
for name, member in inspect.getmembers(_object):
if (
inspect.isfunction(member)
and _object.__qualname__ + "." + member.__name__ == member.__qualname__
):
return inspect.getfile(member)

msg = f"Source for {_object!r} not found"
raise TypeError(msg)


def is_running_interactively():
"""Check if the code is running in an interactive environment."""
try:
from IPython import get_ipython

return get_ipython() is not None
except ModuleNotFoundError:
return False


def build_python_script(component_cls: t.Type[Component]) -> str:
"""Build a self-contained python script for the provided component class, which will act as
the `src/main.py` script to execute the component.
Expand All @@ -281,7 +312,23 @@ def build_python_script(component_cls: t.Type[Component]) -> str:
""",
)

component_source = inspect.getsource(component_cls)
if is_running_interactively():
from IPython.core.magics.code import extract_symbols

component_source = "".join(
inspect.linecache.getlines( # type: ignore[attr-defined]
new_getfile(component_cls),
),
)
component_source = extract_symbols(
component_source,
component_cls.__name__,
)
component_source = component_source[0][0]

else:
component_source = inspect.getsource(component_cls)

component_source = textwrap.dedent(component_source)
component_source_lines = component_source.split("\n")

Expand Down

0 comments on commit 48cd7e0

Please sign in to comment.