-
Notifications
You must be signed in to change notification settings - Fork 448
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use variables in %%configure #834
Comments
🤖 Thank you for your feature request. I understand that you want to enhance the Currently, the Here is an example of how you could modify the @magic_arguments()
@cell_magic
@wrap_unexpected_exceptions
@handle_expected_exceptions
@_event
def configure(self, line, cell="", local_ns=None):
try:
# Evaluate Python expressions in the input string
cell = eval(f'f"""{cell}"""')
dictionary = json.loads(cell)
except ValueError:
self.ipython_display.send_error(
"Could not parse JSON object from input '{}'".format(cell)
)
return
args = parse_argstring_or_throw(self.configure, line)
if self.session_started:
if not args.force:
self.ipython_display.send_error(
"A session has already been started. If you intend to recreate the "
"session with new configurations, please include the -f argument."
)
return
else:
self._do_not_call_delete_session("")
self._override_session_settings(dictionary)
self._do_not_call_start_session("")
else:
self._override_session_settings(dictionary)
self.info("") With this modification, you can use Python expressions in the %%configure
{
"executorCores": os.environ['EXECUTOR_CORES'],
"spark.sql.some.key": get_some_key()
} In this example, Please note that this modification could potentially introduce security risks, as it allows execution of arbitrary Python code. Make sure to only use it in a trusted environment. I hope this helps! If you have any other questions or need further clarification, feel free to ask. SourcesThis response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot. |
I think we also need to enhance |
🤖 Hello, Thank you for your detailed feature request. I understand that you want to enhance the To achieve this, you can modify the Here is a modified version of the @magic_arguments()
@cell_magic
@argument(
"-f",
"--force",
type=bool,
default=False,
nargs="?",
const=True,
help="If present, user understands.",
)
@wrap_unexpected_exceptions
@handle_expected_exceptions
@_event
def configure(self, line, cell="", local_ns=None):
try:
# Evaluate the cell content as Python code
dictionary = eval(cell, local_ns)
except Exception as e:
self.ipython_display.send_error(
"Could not evaluate cell content as Python code: {}".format(e)
)
return
args = parse_argstring_or_throw(self.configure, line)
if self.session_started:
if not args.force:
self.ipython_display.send_error(
"A session has already been started. If you intend to recreate the "
"session with new configurations, please include the -f argument."
)
return
else:
self._do_not_call_delete_session("")
self._override_session_settings(dictionary)
self._do_not_call_start_session("")
else:
self._override_session_settings(dictionary)
self.info("") With this modification, you can use Python expressions in the %%configure
{
"executorCores": get_executor_cores(),
"spark.sql.some.key": os.environ["SOME_ENV_VAR"]
} In this example, Please note that this modification could potentially introduce security risks, as it allows execution of arbitrary Python code. Make sure to use it in a trusted environment. I hope this helps! If you have any further questions, feel free to ask. Best, SourcesThis response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot. |
@sunayansaikia FWIW, one-workaround is to write a function to modify the global sparkmagic |
Is your feature request related to a problem? Please describe.
Currently, in the Pyspark kernel wrapper, we can define
%%configure
as described below in a notebook cell -However, it seems, it does not allow me to pass a value dynamically at runtime; e.g. in the above configuration, I'd like to be able to have the 'executorCores' and 'spark.sql.some.key' set dynamically; e.g.: may be to something that a function would return.
Describe the solution you'd like
Would it be possible to enable somethig like below (example) or may be something better?
Describe alternatives you've considered
No alternative solution found
Additional context
No additional context
The text was updated successfully, but these errors were encountered: