Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parameterized notebook examples #108

Merged
merged 21 commits into from
Jun 5, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
ac26c38
Added batched_example/GSSHA_Workflow_Batched_Example1.ipynb
jlstevens May 30, 2018
910be93
Updated parameter precedences for parambokeh compatibility
jlstevens May 30, 2018
59fa600
Stripped notebook metadata
jlstevens May 30, 2018
0d20ac7
Added nbconvert report generation example
jlstevens May 30, 2018
dc3d043
Set a more appropriate rain_duration for viewing results
jlstevens May 31, 2018
014a634
Simplified nbconvert command in first example notebook
jlstevens May 31, 2018
b19c86d
Added global_params helper function to earthsim
jlstevens May 31, 2018
daeef8b
Added GSSHA_Workflow_Batched_Example2.ipynb demonstrating global_params
jlstevens May 31, 2018
9a3a865
Prototype param command for setting global_params
jlstevens Jun 1, 2018
095c259
Registered entry point for the console script
jlstevens Jun 1, 2018
21087d9
Updated instructions in the second example notebook
jlstevens Jun 1, 2018
a0d576b
Added Rain_Intensity_Sweep_Example.ipynb
jlstevens Jun 1, 2018
6eb677a
Skipping RainIntensity_Sweep_Example notebook in tests
jlstevens Jun 1, 2018
6547fb2
Updated regular expression used to exclude lancet notebook
jlstevens Jun 1, 2018
8a48011
Removed redundant/duplicate skip_run list, and added missing undersco…
ceball Jun 1, 2018
14850f1
Added lancet to the install requires
jlstevens Jun 4, 2018
5361524
Added lancet to environment.yml
jlstevens Jun 4, 2018
6a3e26f
Update GSSHA_Workflow_Batched_Example2.ipynb
jbednar Jun 4, 2018
a370b92
Fixed typos in notebook
jbednar Jun 4, 2018
46c21c8
Renamed earthsim.global_params to earthsim.parameters
jlstevens Jun 4, 2018
9e38275
Using user supplied name if available for parameters
jlstevens Jun 5, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions earthsim/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
import param

def params_from_kwargs(**kwargs):
"""
Utility to promote keywords with literal values to the appropriate
parameter type with the specified default value unless the value is
already a parameter.
"""
params = {}
for k, v in kwargs.items():
kws = dict(default=v)
if isinstance(v, param.Parameter):
params[k] = v
elif isinstance(v, bool):
params[k] = param.Boolean(**kws)
elif isinstance(v, int):
params[k] = param.Integer(**kws)
elif isinstance(v, float):
params[k] = param.Number(**kws)
elif isinstance(v, str):
params[k] = param.String(**kws)
elif isinstance(v, dict):
params[k] = param.Dict(**kws)
elif isinstance(v, tuple):
params[k] = param.Tuple(**kws)
elif isinstance(v, list):
params[k] = param.List(**kws)
elif isinstance(v, np.ndarray):
params[k] = param.Array(**kws)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd greatly prefer this to be done using a lookup table. Something like:

param_types = {
    bool:	param.Boolean,
    int:	param.Integer,
    float:	param.Number,
    str:	param.String,
    dict:	param.Dict,
    tuple:	param.Tuple,
    list:	param.List,
    np.ndarray:	param.Array,
}


def params_from_kwargs(**kwargs):
    params = {}
    for k, v in kwargs.items():
        kws = dict(default=v)
        if isinstance(v, param.Parameter):
            params[k] = v
        elif v in param_types:
            params[k] = param_types[t](**kws)
        else:
            params[k] = param.Parameter(**kws)
    return params

That way other types could be registered by adding them to this dictionary.

Copy link
Contributor Author

@jlstevens jlstevens Jun 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is code I've used in at least three separate contexts, once in another project and once in holoviews itself. I would like to see it moved to param instead of duplicating it in different places. It is often useful!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy for it to move to param, though if it's done there, it will need a good description mentioning the limitations (i.e. that only a few Parameter types will ever be supported in this way).

Copy link
Contributor Author

@jlstevens jlstevens Jun 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll submit a PR to param for this code but I don't think this PR should wait on that. Once this is in param, I can update earthsim to import it accordingly.

else:
params[k] = param.Parameter(**kws)
return params


def parameters(**kwargs):
"""
Utility to easily define a parameterized class with a chosen set of
parameters specified as keyword arguments. The resulting object can
be used to parameterize a notebook, display corresponding widgets
with parambokeh and control the workflow from the command line.
"""
name = kwargs.get('name', 'Parameters')
params = params_from_kwargs(**kwargs)
params['name'] = param.String(default=name)
return type(name, (param.Parameterized,), params)
43 changes: 43 additions & 0 deletions earthsim/__main__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
"""
Prototype of a param script that can set global_params via the
appropriate environment variable to execute an arbitrary command.

Example usage:

param -cmd 'jupyter nbconvert --execute GSSHA_Workflow_Batched_Example2.ipynb' -p rain_intensity=25 -p rain_duration=3600
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presumably 'param' is 'earthsim' in this case?

Copy link
Contributor Author

@jlstevens jlstevens Jun 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't sure what to call it: right now it is in earthsim but we hope to migrate it to param. I'm happy to name it whatever you want though I don't know if an earthsim console script would have a long term future/purpose.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see -- I thought the command name would default to the module name, but I see later that it's explicitly declared param = earthsim.__main__:main. If it can really be supplied as an arbitrary command name, then param seems highly likely to conflict with something else on the user's path, so I think it should be something less generic. Maybe run_with_params? Not sure if running with params is more or less dangerous than running with scissors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe their is a unix/windows command called param so I don't see why param couldn't have a console script of that name. What do you think users might have on their path that conflicts with it?

If we have to worry about a clash, I suppose I might consider something like params or parameters instead.

Copy link
Contributor

@jbednar jbednar Jun 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both of those are equally generic. If the intent is for param to supply this command, then calling it param is reasonable, as having the package param supply the param command won't be surprising to anyone. @ceball, do you see any conflict with some other usage of a param module command? Seems ok to me.

Copy link
Contributor Author

@jlstevens jlstevens Jun 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the intent is for param to supply this command, then calling it param is reasonable

That was what I was thinking at least.

"""

import os
import sys
import json
import argparse
import subprocess
from distutils import spawn

def main():
parser = argparse.ArgumentParser()
parser.add_argument('-cmd',type=str)
parser.add_argument("-p", nargs=1, action='append')
args = parser.parse_args()
if args.p is None:
print('Please supply parameters using the -p flag')
sys.exit(1)

if args.cmd is None:
print('Please supply a command string using the -cmd flag')
sys.exit(1)

execute(args)

def execute(args):
split_strings = dict(el[0].split('=') for el in args.p)
json_dict = {k:eval(v) for k,v in split_strings.items()}
env = os.environ.copy() # Required on windows
env['PARAM_JSON_INIT'] = json.dumps(json_dict)
cmd = args.cmd.split()
p = subprocess.Popen(cmd, env=env,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()

if __name__ == "__main__":
main()
2 changes: 1 addition & 1 deletion earthsim/gssha/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ class Simulation(param.Parameterized):

land_use_grid_id = param.Parameter(default='nlcd',precedence=0.3)

model_creator = param.ClassSelector(CreateModel,default=CreateGSSHAModel(),precedence=0.4)
model_creator = param.ClassSelector(CreateModel,default=CreateGSSHAModel(),precedence=-1)

# TODO: switch to date range...
simulation_start=param.Date(default=datetime(2017, 5 ,9),precedence=0.5)
Expand Down
2 changes: 1 addition & 1 deletion earthsim/gssha/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ class CreateGSSHAModel(CreateModel):
# created or else editing parameters on non-default options is
# confusing.
roughness = param.ClassSelector(RoughnessSpecification,default=UniformRoughness(),doc="""
Method for specifying roughness""")
Method for specifying roughness""", precedence=-1)

out_hydrograph_write_frequency = param.Number(default=10, bounds=(1,60), doc="""
Frequency of writing to hydrograph (minutes). Sets HYD_FREQ card. Required for new model.""", precedence=0.8)
Expand Down
1 change: 1 addition & 0 deletions environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ channels:
- defaults
dependencies:
- python>=3.5
- lancet
- fiona
- rasterio
- gdal
Expand Down
Loading