hk-dev-utils (hkdu)
===================

hk-dev-utils is a set of development utilities for the Heapkeeper project
(http://heapkeeper.org).

SETUP
=====

Put the following lines into your .bashrc:

    export HEAPKEEPER_DEV_DIR=/path/to/heapkeeper
    export HK_DEV_UTILS_DIR=/path/to/hkdu
    export PATH=$HK_DEV_UTILS_DIR/bin:$PATH
    export PYTHONPATH=$HK_DEV_UTILS_DIR/lib:$PYTHONPATH

Perform the following commands:

    $ cd $HEAPKEEPER_DEV_DIR/.git/hooks
    $ ln $HK_DEV_UTILS_DIR/bin/hkdu-pre-commit pre-commit
    $ ln $HK_DEV_UTILS_DIR/bin/hkdu-post-commit post-commit

Other actions:

    - Install gvim.

See the "Debugging" section if something does not work.

SCRIPTS
=======

The bin directory contains bash and Python scripts. The old directory contains
scripts that should be modified to work consistently with the new way of working
of hk-dev-utils.

Generic utilities
-----------------

Generic utilities can be used for any git repository, not just for Heapkeeper.

- bin/hkdu-list-sources: It is similar to "git ls-files", but it only prints
  files that are considered text files. Very useful for grep:

      $ hkdu-list-sources | xargs grep "my pattern"

- old/showcommit: Shows a given commit in a format I like. It does not work now.

Tester scripts
--------------

- bin/hkdu-test: Executes the given tests on the working tree.

- bin/hkdu-testcommit: Executes the given tests on the given revisions. The
  Heapkeeper repository is cloned into a temporary directory and all tests are
  made there so the users can continue their work.

- bin/hkdu-notify: It can be piped after either hkdu-test or hkdu-testcommit to
  give notifications when they are ready. By default it shows a notification
  bubble, and if the test fails, the problems will be shown in an editor window.
  (The editor is gvim by default with the 'q' normal mode command mapped to
  ':q<cr>').

- old/testhtml: Executes different versions of Heapkeeper and compares the HTML
  pages generated by them.

We have the following tests:

- unittest: Runs the unittests of Heapkeeper (src/test.py).
- pylint: Runs pylint on the Python code of Heapkeeper.
- linelength: Checks that no line is longer than 79 characters in the text
  files. (Except for lines that contain the "http://" string.)
- trailingwhitespace: Checks that there is no trailing whitespace.
- trailingline: Checks that there are no empty lines at the end of the files.
- javascript: Starts a hkweb and opens the JavaScript unittest in google-chrome.
  This is the only test with a side-effect; other tests behave nicely and return
  an error report without affecting their environment. Also, this is the only
  test whose result cannot be evaluated automatically. Therefore this is not
  executed by default (all the others are).

Examples:

- Testing the working tree; the results will be printed to the console:

      $ hkdu-test

- Testing the last three commits (two equivalent ways shown):

      $ hkdu-test HEAD^^ HEAD^ HEAD
      $ hkdu-test HEAD~3..HEAD

- Testing all commits between Heapkeeper v0.6 and v0.7:

      $ hkdu-test v0.6..v0.7

- Testing the working tree; we will get back the prompt and show the results in
  a notification bubble and an editor:

      $ hkdu-test | hkdu-notify &

- Testing the last three commits with notifications about the results. If all
  test fails, three graphical editor window will pop up.

      $ hkdu-test HEAD~3..HEAD | hkdu-notify

- Executing only unittest and pylint:

      $ hkdu-test -t unittest:pylint

- Executing everything except unittest and pylint:

      $ hkdu-test -t -unittest:pylint

Hooks
-----

- bin/hkdu-pre-commit: This is a pre-commit git hook. By default, it executes
  the most simple checks (e.g. line length, trailing whitespace) and prevents
  the commit if they fail.

- bin/hkdu-post-commit: This is a post-commit git hook. By default, it hook
  executes all tests (except the javascript one). It reports the result
  asynchronously in a notification bubble, and if the test fails, more verbosely
  in an editor window.

Both hooks can be customized; more information about this can be found is
comments of the text files.

Heapkeeper release scripts
--------------------------

These only need to be known by the person who manages the Heapkeeper homepage and
Heapkeeper releases.

- bin/hkdu-make-package: Makes Heapkeeper packages (tar.gz and zip files).

- bin/hkdu-pushdoc: Pushes the web pages to the Heapkeeper homepage.

- bin/hkdu-pushrelease: Pushes the packages of a new release to the Heapkeeper
  homepage.

LIBRARIES
=========

The lib directory contains Python libraries used by the scripts. They can also
be used directly. I tried to put most of the code here to enhance reusability.

DEBUGGING
=========

Debugging hk-dev-utils is usually easy. You can either use the "-V" option to
make the scripts very verbose, or add printouts to the code. (See the "logging"
section for more information on this.) The exceptions are printed to the
console as stack traces.

When debugging is hard
----------------------

I said "usually", because there is one exception: the post commit hook. In the
post commit hook, I use a dirty trick I don't even understand; I use gvim to
start a new process, because otherwise interactive rebase will wait for the
popped up result window to be closed in each step. If you have a problem with
the commit hook, modify the source code to use "&" to start a new process
instead of gvim. (See more information about why this is necessary in the
comments of the source code of the post hook.)

Also, there is another trick that you have to know when the post commit hook
fails. The post commit hook uses a lock file (~/.hk-dev-utils/repository.lock)
to ensure that the prompt is not given back until the original Heapkeeper
repository has been cloned. The post commit hook creates the lock file and
supplies the --remove-lock parameter to the hkdu-testcommit script asking it to
remove the lock file after it had finished the cloning. If the hkdu-testcommit
script is not found in the PATH or it fails and exits very early (before doing
the cloning and removing the lock file), the post hook will wait forever after
printing "hkdu: tests started...", because it waits for the lock file to be
removed, which will never happen. If this is the case, you can manually remove
the lock file. Afterwards hkdu-post commit will print "hkdu: test moved into the
background.", but that only means that it realized that the lock file is no
longer there and exited. (See more information about how the locking works and
why it is necessary in the comments of the source code of the post hook.)

Logging
-------

The Python `logging` library is used extensively for logging. The command line
utilities have the same switches to set the logging level. The levels that
correspond with the levels of the Python logging library are the following:

  1. quiet: nothing is printed, not even errors
  2. no-warnings: only errors are printed
  3. normal: errors and warnings are printed
  4. verbose: info printouts are also printed
  5. very verbose: even debug printouts are printed

OTHER ASPECTS
=============

- Help: all scripts accept the "--help" option that prints their description and
  all their options.

- ~/.hk-dev-utils: This directory is created when needed.

- Temporary directories: Temporary directories created by tests are not
  currently deleted. Temporary directories are best to be stored on the same
  file system partition as the main git repository, because that way cloning
  the repository can be faster and less space-consuming (git can trick with
  hardlinks). The $TMPDIR environment variable can be used to set this.

- Settings: customization is quite ad-hoc. Sometimes information is read from
  environment variables (like $HEAPKEEPER_DEV_DIR), sometimes it is read from a
  command-line option and stored as a global variable of hkdu_utils (like
  SHA_LENGTH).

- Initialization: initialization is also quite ad-doc. There is a check-egg
  problem between logging the command line paramater parsing errors and using
  the command line parameters to set the logging level. The timing of teading
  the value of the HEAPKEEPER_DEV_DIR is also not trivial, because on one hand,
  we have to read it before importing hkutils (the path has to be set), on the
  other hand, the HEAPKEEPER_DEV_DIR variable can be bad in several ways, so it
  would be good to log it, which means that it has to be started after the
  logging, which is after parsing the command line arguments, which is of course
  started after the import statements.

- Using Heapkeeper modules: hk-dev-utils uses some utility functions from
  Heapkeeper. When Heapkeeper had a function I needed, I did not want to copy
  the code, so I just imported it. hkdu_utils adds the $HEAPKEEPER_DEV_DIR/src
  directory to sys.path to make this possible. This fact the hk-dev-utils uses
  Heapkeeper means that we have to be careful with these dependencies (e.g. we
  cannot just remove them from Heapkeeper without deprecating them through a
  release).

- Compatibility between hk-dev-utils and Heapkeeper versions: hk-dev-utils does
  not have releases and version numbers. The The compatibility between
  different versions of hk-dev-utils and Heapkeeper is described in
  doc/heapkeeper-compatibility.txt.

TODO
====

Possible improvements:
- Examine this "git rebase works asynchronously only if processes in the post
  hook are started via gvim" problem.
- Rewrite the release scripts and the scripts in the old/ directory.
- The notifications could display the problematic commit and tests.
- hkdu-list-sources could use the "file" unix command to decide whether a file
  is text file or not.
- hkdu-test and hkdu-test-commit could remove all *.pyc files before each test
  execution.
- The temporary directory could be removed after hkdu-test-commit. Probably the
  "--only-refs" option should be modified or removed in this case.
- We could use a new environment variable (e.g. HKDU_TMP_DIR) to contain the
  temporary directories.
- Make the javascript test get the name of the browser from an environment
  variable.
- Create a test that tests that the commit messages have a correct topic.