Python Module Development

Setting Up a Python Module for development

To build and install python modules, we use setup.py. It is always recommended to do this in a virtual environment (see using conda).

To setup the module for interactive development, we call

$ python setup.py develop   # build the module locally and allow quick updating

This is recommended.

To install the module once all the module files are finalized, we call

$ python setup.py install   # install the package for long-term use (unrecommended)

This is unrecommended, as changes in the src files will not update the installed module.

To remove the python module from the path, we call

$ python setup.py develop --uninstall

when using the develop option.

Testing a Python Module

Use Python's unittest module to write unit tests. Write tests in the /test directory. See /test/rect_test.py for more details.

Run tests via nosetests from the terminal.

Profiling a Python Module

From within IPython, define a funcion to_profile() with the code of interest. Then consider running the magic %prun after importing the cProfile module

import cProfile
%prun -D output_dump_location to_profile()

For visualization of cProfile output, use kcachegrind and pyprof2calltree from the terminal

$ pyprof2calltree -i output_dump_location -k

For line-by-line profiling and memory profiling, use the following two extensions to IPython

load_ext line_profiler
load_ext memory_profiler

Then use the magic functions %lprun and %mprun on to_profile() within IPython.

Both line_profiler and memory_profiler must be installed using pip. Don't forget to add -f or -m flags to indicate which function or modules to profile or the -T flag to print text output to file.

See http://scikit-learn.org/stable/developers/performance.html for more details.