Skip to content

Github mirror of "analytics/refinery" - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for contributing

Notifications You must be signed in to change notification settings

wikimedia/analytics-refinery

Repository files navigation

Refinery

Refinery contains scripts, artifacts, and configuration for WMF's analytics cluster.

Setting up the refinery repository

  1. Install git-lfs on your system. Debian includes a git-lfs package.

  2. Make sure the docopt and dateutil Python packages are available on your system.

    On Ubuntu systems, you can achiev this by running

    sudo apt-get install python-docopt
    sudo apt-get install python-dateutil
    
  3. Clone the repository.

    You can find the commands to clone the repository at WMF's gerrit.

    To clone anonymously, just run

    git clone https://gerrit.wikimedia.org/r/analytics/refinery
    
  4. change to the cloned repository by running

    cd refinery
    
  5. Initialize git-lfs by running

    git lfs install
    
  6. Pull existing artifacts into the repository by running

    git lfs pull
    

    (Depending on you internet connection, this step may take some time.)

  7. Add the refinery/python directory to your PYTHONPATH.

    To add it only in the running shell, you can use

    export PYTHONPATH=/path/to/analytics/refinery/python
    

    Please refer to your operating system's documentation on how to do this globally.

  8. Done.

Oozie job naming convention

  • Job base names is following directory pattern in the oozie directory, replacing slashes with dashes. For instance webrequest/load/bundle.xml job is named webrequest-load-bundle, and last_access_uniques/daily/coordinator.xml is named last_access_uniques-daily-coord.
  • Root job names end either in -bundle or -coord, while children job names end with job parameters separated with dashes.

About

Github mirror of "analytics/refinery" - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for contributing

Resources

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published