This repository contains the frontend website for ARG-V. For the transformer, see: unl-pal/argv-transformer
ARG-V front-end UI repository using the Django web framework. This repository also contains several back-end scripts.
- Python 3.5+
- pip
See requirements.txt for full list of Python requirements. Note development and production environments have different requirements files (see the requirements directory). By default, production is used.
- Install
python3 - Install
pip3 - Open terminal in location you want to install the website
- Clone repository from GitHub (this creates folder named
argv-www) - Navigate terminal to root of project directory (where
manage.pyis):cd argv-www - Use command
pip3 install -r requirements.txtto install all default requirements - Generate a secret key and copy to the clipboard:
python manage.py shell -c "from django.core.management import utils; print(utils.get_random_secret_key())" - Copy
env.templateto a file named.envand edit for your local configuration - Create migration files for database:
python3 manage.py makemigrations website backend - Create database tables:
python3 manage.py migrate
- Open terminal to where
manage.pyis located - Run command:
manage.py install boato install the Boa backend. In the future,boacan be replaced with the name of other backends. - Run command:
manage.py enable boato enable Boa's filters on the web server - When creating a new project selector in the UI, you should see the installed filters as options in the dropdown menus.
- Open terminal to where
manage.pyis located - Run command:
manage.py install argvto install the ARG-V backend. In the future,argvcan be replaced with the name of other backends. - When creating a new project transformer, you should see the ARG-V transformer as an option.
The backend has several scripts that need to run. Be sure each of them is running for proper functioning.
- Open terminal in the location of
manage.py - Run command:
python3 manage.py rundiscovery
- Open terminal in the location of
manage.py - Run command:
python3 manage.py runcloner
- Open terminal in the location of
manage.py - Run command:
python3 manage.py runfilters
- Open terminal in the location of
manage.py - Run command:
python3 manage.py runtransforms
- Create an admin user:
python3 manage.py createsuperuser - Create static files:
python3 manage.py collectstatic - Load system data:
python3 manage.py loaddata --app website basedata - Run django's development server using command:
python3 manage.py runserver
Currently, there is support for the Boa backend. In the future we hope to also include support for other backends. Backends are used to run queries and return lists of matched projects to be cloned. Backends consist of a poller class which inherits from the poller found in the backend app in the ARG-V repository. Pollers trigger every several seconds, and query the database for any new project selectors that have yet to be processed. Backends contain a list of filters that can be enabled on the web server when creating project selectors.
Currently, we have automatic support for a few transformers. Right now, the transformations to apply to each project are created automatically, however in the future we would like users to be able to specify different transformations to for different project selections. The transformations are installed just like filters for selectors, using various backends that users deploying ARG-V will be able to swap in and out eventually.
The poller runs every few seconds and inherits from a base class in the ARG-V repository. The poller checks the database for unprocessed project selectors, then generates a query based on the filters listed under the selector. The query is then sent to the appropriate backend, which returns a list of discovered projects.
The poller runs every few seconds and inherits from a base class in the ARG-V repository. The poller checks the database for unprocessed project selectors, then generates a query based on the filters listed under the selector. The query is then sent to the appropriate backend, which returns a list of discovered projects.
The cloner runs every few seconds and looks for new urls that were discovered but not yet cloned. It then uses the input dataset from the original selector to search for each project and clones them. After completion of its activities, it sets the url's host and path name so other services can find the location in the cluster of the cloned project.
The transformer runs every few seconds and processes downloaded projects. It uses the set host and path name to find each project directory, then transforms the project source code using the default installed transformations. It then saves an entry in the database for this fully processed project which is used for zipping the transformed project and downloading it.
If you're using Ubuntu + Apache we recommend using this guide: https://www.digitalocean.com/community/tutorials/how-to-serve-django-applications-with-apache-and-mod_wsgi-on-ubuntu-16-04 to help set up Django on your server
/data
/repos
...
/transformed
/selector/pk
/TransformOption.pk
...
/TransformOption.pk
...
/transform/pk
/TransformOption.pk
...
/TransformOption.pk
...