Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Use cloudpickle to run remote functions in jupyter environments #856

Merged
merged 18 commits into from
Jan 17, 2025

Conversation

dleviminzi
Copy link
Collaborator

@dleviminzi dleviminzi commented Jan 14, 2025

Resolve BE-2218

This PR changes the strategy used for running remote functions in notebooks. Instead of creating a temporary file built by stitching together executed cells, we use cloudpickle to capture the required function + deps. This resolves issues with re-executed cells and is generally more reliable.

This PR also changes the way that the default python environment is selected for Image. It tries to see if the active python version from the local environment is a valid remote python version. If it is, it will use that version by default. I can remove this change or limit it to ipython environments only. This is nice when working in jupyter notebooks because cloudpickle dumps are dependent on the python version. If you create a dump in 3.12 and try to load it in 3.10, things will not work.

TODO:

  • Test whether different protocols improve compatibility for cloudpickle python version mixing
  • Alert users when they try to run different local/remote python versions in jupyter environments
  • Explore ways to detect function content changes to avoid re-syncing when the cell has been re-executed but no code has changed

pkg/gateway/services/stub.go Outdated Show resolved Hide resolved
sdk/src/beta9/abstractions/base/runner.py Outdated Show resolved Hide resolved
sdk/src/beta9/abstractions/base/runner.py Outdated Show resolved Hide resolved
sdk/src/beta9/abstractions/base/runner.py Show resolved Hide resolved
sdk/src/beta9/abstractions/base/runner.py Outdated Show resolved Hide resolved
sdk/src/beta9/abstractions/image.py Show resolved Hide resolved
sdk/src/beta9/env.py Outdated Show resolved Hide resolved
sdk/src/beta9/runner/common.py Outdated Show resolved Hide resolved
sdk/src/beta9/runner/common.py Outdated Show resolved Hide resolved
sdk/src/beta9/runner/common.py Outdated Show resolved Hide resolved
sdk/src/beta9/runner/common.py Outdated Show resolved Hide resolved
@luke-lombardi luke-lombardi merged commit 3722545 into main Jan 17, 2025
3 checks passed
@luke-lombardi luke-lombardi deleted the dlm/better-noteboook-support branch January 17, 2025 23:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants