Skip to content

Developer's Guide

Jump to bottom Edit New page
andresriancho edited this page Mar 27, 2013 · 31 revisions

Introduction

This page is the most important source of information for w3af developers and contributors. This documents has links to the development process, code conventions, tips and tricks, etc.

There are basically two different types of w3af developers:

  • Code Hackers: Advanced users that might identify bugs or potential improvements and simply modify a couple of files to apply their change. These users might be new to git, w3af and python. We recommend they use the Contributing 101 guide until you feel comfortable with the process.

  • Developers: Advanced users which are interested in enhancing the framework in more complex ways, add new features, refactor sections of code, etc. We recommend you follow this document in order to setup your development environment.

Building your development environment

IDE

Each developer can choose his own IDE to work with w3af, but we're using Eclipse with PyDev for framework development. Please follow the steps in the Eclipse, PyDev and EGit setup to get your Eclipse environment working with w3af in order to have code completion for w3af classes, references (ctrl + click over a class), fast code search with (ctrl+shift+R) and all the other Eclipse goodies.

Extra tools

pylint is a code verifier for Python. All pull requests, patches, etc. which are submitted to us via github 'must be reviewed by pylint first'. In all systems with pip installed you can install get the latest pylint by running:

sudo pip install pylint --upgrade

We recommend you use pip instead of apt-get install pylint because you'll get a more recent version of the tool. And then run the following command:

pylint --additional-builtins=_ -E plugins/audit/new_module_you_developed.py

Unit testing

Nosetests is the best unittest runner we've found, it's extensible using plugins and fits all of our needs. We're also using the python mock library in some tests and a set of plugins to enhance nose's output and provide code coverage. Installation of these tools is straight forward:

sudo pip install mock
sudo pip install pylint
sudo pip install nose
sudo pip install nose-exclude
sudo pip install yanc
sudo pip install coverage
sudo pip install sneazr-pynotify
sudo pip install -e git+git://github.com/andresriancho/nose-timer.git#egg=nose-timer
sudo pip install psutil

Once again, please note that we use mock for writing our unittests that require mocked objects, keep that in mind when writing new unittests!

You can run all of w3af's unittests with the following command:

cd w3af/
nosetests --with-yanc --with-doctest --doctest-result-variable=_abc_ --doctest-tests \
          --exclude-dir-file=exclude_dirs.txt --with-sneazr-pynotify --with-timer

This is the content of exclude_dirs.txt:

core/ui/gui/
plugins/discovery/oHalberd

Alternatively, you can download the nose.cfg file from our w3af-qa repository, save it to the project root and run the tests using this shorter syntax:

wget https://raw.github.com/andresriancho/w3af-qa/master/nose/nose.cfg # You only need to run this once
nosetests --config=nose.cfg

If you want to measure test coverage, you can also use nosetests:

nosetests --config=nose.cfg --with-coverage --cover-package=core.data.parsers core/data/parsers/

In order to run only the smoke tests you can run:

nosetests --config=nose.cfg -a smoke core/data/parsers/

And if you want to only run the test cases that failed in the previous run, which is perfect for running until all cases in a directory pass and then run all the tests again:

nosetests --config=nose.cfg --failed --with-id core/data/parsers/

TDD freak? Then you'll want to install the randomize plugin (read this) and run your tests with --randomize.

Running the test suite

Our test suite requires a running moth VM to have a set of vulnerable web applications. For setting up this VM just follow these steps:

  1. Install a vanilla Ubuntu server 12.04
  2. Clone moth's repository into /var/www/
  3. Setup apache to use the configuration available in moth's repo to have the virtual hosts setup
  4. In your workstation setup /etc/hosts to point moth to the IP address of the virtual system

Continuous integration

We're using Jenkins-CI for continuous integration. The CI server is still being built and kind of unstable.

Debugging

One of the neatest tricks for debugging w3af is to run this command in a console:

HTTP_PROXY=127.0.0.1:8080 nosetests --config=nose.cfg plugins/tests/attack/test_rfi.py

Which will route all HTTP traffic through a local proxy (burp, zap, etc.) which will allow you to view a very clear log of what w3af is sending to the wire.

Code conventions

PEP8 MUST be used for all new code. Also, if you're working on old code and see any non-PEP8 code snippets, update it!

Dependencies and third party libraries

Keep in mind that we don't usually add new dependencies to the project, and if we do we need to verify the following:

  • The license is GPL compatible
  • The library is stable and maintained
  • Unless there is no other way around it, we recommend using pure-python libraries.
  • Since this changes the packaging of w3af, please ask in w3af-develop about adding a new dependency
  • If a new dependency is added you need to check for correct installation at the dependency_check.py files. There is one file that checks w3af-wide dependencies and another that checks the dependencies for the GTK user interface. Make sure you modify the corresponding file.

SDLC

Our software development life cycle uses A successful Git branching model (also available here in PDF format) and git-flow.

Getting started

For the best introduction to get started with git flow, please read Jeff Kreeftmeijer's blog post:

http://jeffkreeftmeijer.com/2010/why-arent-you-using-git-flow/

Or have a look at one of these screen casts:

Forking

Create a fork by clicking here. Clone w3af's fork from your github user URL and call it w3af_<user>. Make sure you replace both instances of <user> in the next line:

git clone [email protected]:<user>/w3af.git w3af_<user>

Keep your repository updated

If you want to contribute with the w3af project the best way is to follow the fork and pull-request methodology. If you're not a developer this basically means you'll create a copy of w3af's repository, apply your changes, and then send your changes back to us using a pull-request (as explained here)

After you fork our repository we'll continue to work on it, so your copy will get outdated. In order to keep your repository updated, please follow these steps.

Set the remote upstream repository to w3af's main repo

git remote add upstream [email protected]:andresriancho/w3af.git

Retrieve the updates from the upstream repository

git fetch upstream

Make sure that you know what you're getting from upstream and what is in your master

git log upstream/master ^master
git log ^upstream/master master

If everything is good, you can merge upstream into your clone

git merge upstream/master

Now you're ready to update your fork

git push origin master

Sending pull requests

Before you submit a pull request please make sure:

  • You're submitting it against the development branch.
  • Provide a good description of what your doing and why.
  • Provide tests that cover your changes.
  • Run the unittests locally first and make sure they all PASS.

Branches

Only contributors with write access to the w3af repository can create branches. We encourage the creation of branches for any feature development using git flow. Please keep your branch updated with the development branch!

GIT tricks

Git extensions

git-extras provides many interesting shortcuts for the developer. Some of them are:

  • git-local-commits: List all commits on the local branch that have not yet been sent to origin.
  • git-undo: Remove the latest commit
  • git-changelog: Generate a changelog file
  • git-release: Release a new version of the software

git-flow provides very helpful extensions for our SDLC.

.gitconfig

.gitconfig is your friend, here are some aliases we recommend (please make sure you collapse long lines into one line after copy+paste):

[alias]
lg = log --graph --all --format=format:'%C(bold blue)%h%C(reset) - %C(bold green)(%ar)%C(reset) 
     %C(white)%s%C(reset) %C(bold white)— %an%C(reset)%C(bold yellow)%d%C(reset)' --abbrev-commit
     --date=relative

After that git lg should show you an easy to read, graphical output.

Git commands

  • git rev-parse HEAD: returns the latest commit
  • git reset --hard <commit-id>: Moves the current working copy to commit-id and removes all childs from that commit from the index (git log)
  • git for-each-ref: Show all commit-ids for the latest commit in each tag, branch, etc.

Releasing a new version of w3af

Releasing a new version of w3af is much more than just building the new "bz2 package"! For instructions about the technical and community tasks around a release, please visit our shipping a new release page.

Add a custom footer
Add a custom sidebar
Clone this wiki locally