-
Notifications
You must be signed in to change notification settings - Fork 791
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feature: fast initializer for conda environments #2226
Conversation
metaflow/metaflow_config.py
Outdated
@@ -433,6 +433,12 @@ | |||
# should result in an appreciable speedup in flow environment initialization. | |||
CONDA_DEPENDENCY_RESOLVER = from_conf("CONDA_DEPENDENCY_RESOLVER", "conda") | |||
|
|||
# Conda Fast init binary url | |||
CONDA_FAST_INIT_BIN_URL = from_conf("CONDA_FAST_INIT_BINARY_URL", None) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you remove this config and introduce the path to actual binary in the install_fast_initializer
method instead? also, if you can note some latency improvements in the PR, that will be great!
looks good! can you verify that this works with dev-gcp and dev-azure too before merging? |
adds a feature flag to switch between regular micromamba and a custom binary for bootstrapping of environments when executing remotely.
also adds deferring of micromamba install until the binary location is actually requested.
some benchmarking results with
@pypi