Script and Job Management

This page documents how to combine shell scripts and python scripts to run multiple trials of similar experiments.

Overview of the workflow

Typically in research, we need to run multiple trials of the same experiments with either the same settings/arguments (e.g. replications) or different settings/arguments (e.g. tuning/robustness). To reduce the amount of duplicated code, this page's recommended workflow is

• Code a single 'experiment' python script that takes arguments
• Test the 'experiment' script
• Code a 'script builder' python script that creates a shell script to call the 'experiment' script with appropriate settings
• Use the 'script builder' to create shell scripts
• Call the shell scripts to run multiple trials of the same experiment.

The 'experiment' script takes in inputs and saves results to specified output locations. The 'script builder' creates a shell script that repeatedly calls the 'experiment' script.

An alternative approach is to have a single python script, that consists of a for-loop around the experiment code. Although this is a good quick and dirty solution, it is not flexible enough to allow running experiments in parallel or on cluster managers, which prefer 'jobs' to be run from the command line.

Single Experiment Script

The experiment script will consist mainly of three steps

1. Parse command-line arguments
2. Run 'main' experiment script
3. Save Outputs/Figures

Using logging

To Do. Highly recommended.

Using argparse

To vary the parameters for each experiment, we pass arguments to a python script from a shell (e.g. python <script_name>.py <args>). The classic built-in python way to pass arguments is sys.argv. The argparse library is a utility module which parses sys.argv, generates help and usage documentation, and raises errors for invalid arguments.

To use argparse, first construct an ArgumentParser then use the method function parse_args(). Here's a short example

my_parser = argparse.ArgumentParser(
    description="Short Description",
    fromfile_prefix_chars='@',
    )
my_parser.add_argument("--float", default=1.0, type=float,
    help="help string for argument --float")
my_parser.add_argument("--int", default=0, type=int)
my_parser.add_argument("--string", default="")
my_parser.add_argument("--bool", default=True, type=_str_to_bool)

Note that fromfile_prefix_chars allows arguments to be passed from a text file <options> by passing @<options>. To pass in a boolean as an argument, either pass it as an int and cast or use the following helper _str_to_bool converter

def _str_to_bool(s):
    """Convert string to bool (in argparse context)."""
    if s.lower() not in ['true', 'false']:
        raise ValueError('Need bool; got %r' % s)
    return {'true': True, 'false': False}[s.lower()]

Script Builder

To Do