diff --git a/README.md b/README.md index 190fc4d0d..5bd178c64 100644 --- a/README.md +++ b/README.md @@ -5,144 +5,69 @@ Internet of Good Things ([IoGT][2]) is developed as a public good under a [BSD-2 The development uses the Python programming language, building on the [Django][4] server-side web framework combined with the [Wagtail][5] content management system. In line with the latest Wagtail [Long Term Support release][6], IoGT 2.x supports: -- Wagtail 2.11.x -- Django 2.2.x, 3.0.x and 3.1.x -- Python 3.8 and 3.9 +- Wagtail 2.15.x +- Django 3.0.x, 3.1.x and 3.2.x +- Python 3.8, 3.9 and 3.10 - PostgreSQL, MySQL and SQLite as database backends ## Getting started -If you'd like to contribute to IoGT development, the best way to get started is to clone a copy of this repository and run the code on your machine. +The easiest way to get started is to run IoGT using Docker. Make sure you have Docker Desktop installed, or at the very least the Docker Engine and Docker Compose (which are included in Docker Desktop). + +Check out the code. ``` git clone https://github.com/unicef/iogt.git cd iogt ``` -Create an [isolated Python environment][1], into which we will install the project's dependencies. This is highly recommended so that the requirements of this project don't interfere with any other Python environments you may have in your system, including the global Python environment. We also recommend that you create a new environment outside of the project directory. +Create the database. By default, an SQLite database will be created. This will take a few minutes on the first run because the IoGT container image needs to be created from scratch. ``` -python3 -m venv -source /bin/activate -pip install -r requirements.txt -``` - -Create the database. By default, an SQLite database will be created. -``` -./manage.py migrate +docker compose run --rm django python manage.py migrate ``` Create a superuser account for administration purposes. ``` -./manage.py createsuperuser +docker compose run --rm django python manage.py createsuperuser ``` Compile .po language files stored in locale/ ``` -./manage.py compilemessages +docker compose run --rm django python manage.py compilemessages ``` -Finally, start the server. +Start the server. ``` -./manage.py runserver +docker compose up -d ``` -Once running, navigate to http://localhost:8000 in your browser. - - -### Running ElasticSearch (Optional) - -1. Set up an elastic search cluster -2. Update local.py to use Elasticsearch as the backend. More details [here](https://docs.wagtail.io/en/stable/topics/search/backends.html#elasticsearch-backend) - ``` - WAGTAILSEARCH_BACKENDS = { - 'default': { - 'BACKEND': 'wagtail.search.backends.elasticsearch7', - 'URLS': ['http://localhost:9200'], - 'INDEX': 'iogt', - 'TIMEOUT': 5, - 'OPTIONS': {}, - 'INDEX_SETTINGS': {}, - 'AUTO_UPDATE': True, - 'ATOMIC_REBUILD': True - } - } - ``` - -3. Run `./manage.py update_index` to update the ElasticSearch Index - -### Setting up test data - -It is possible to automatically populate the database with example data for a basic test site. -``` -./manage.py create_initial_data -``` - -Optionally, create the main menu automatically as well. +Check the logs. ``` -./manage.py autopopulate_main_menus +docker compose logs -f ``` +Once running, navigate to http://localhost:8000/ in your browser. Log in to the admin UI at http://localhost:8000/admin/ - use the superuser name and password you created earlier. -### Running Tests +Shut everything down. ``` -./manage.py test --settings=iogt.settings.production +docker compose down ``` -## Setup with Docker Compose - -You can choose to set up the project locally using Docker Compose. This setup is recommended if you want to replicate the production environment. - -### Steps - -Clone the repository - -``` -git clone https://github.com/unicef/iogt.git -``` -Run setup command. This will set up the database and also update the ElasticSearch Index. -``` -cd iogt -make setup -``` - -Create a super user account for administration purposes. -``` -docker-compose run django python manage.py createsuperuser -``` - -Run the compose file -``` -make up -``` -You're all set now. See the `Makefile` for other commands related to docker-compose +# Optional ## Setting up test data It is possible to automatically populate the database with example data for a basic test site. -``` -docker-compose run django python manage.py create_initial_data -``` -Optionally, create the main menu automatically as well. -``` -docker-compose run django python manage.py autopopulate_main_menus +```sh +docker compose run --rm django python manage.py create_initial_data ``` -## Running Tests - -Run the following commands: -``` -make tests -make selenium-test -``` +Optionally, create the main menu automatically as well. -In parallel, for example, with 4 processes: -``` -IOGT_TEST_PARALLEL=4 make test -IOGT_TEST_PARALLEL=4 make selenium-test +```sh +docker compose run --rm django python manage.py autopopulate_main_menus ``` -More details of the Selenium tests can be found in the [Selenium test README][9]. - ## Configuring the Chatbot Follow instructions [here](messaging/README.md) @@ -182,12 +107,32 @@ Even though it has a column "is in use", its data is currently not updated autom Page caching is provided by Wagtail Cache and is deactivated by default. To start caching pages, follow the [activation intructions][7]. +## Elasticsearch + +To use Elasticsearch as a search backend instead of the relational database see the [setup instructions][11]. + ## Migrating content from IoGT v1 Follow instructions [here](iogt_content_migration/README.md) ## Add/Update Package Follow instructions [here][10] +# Running Tests + +Run the following commands: +``` +make tests +make selenium-test +``` + +In parallel, for example, with 4 processes: +``` +IOGT_TEST_PARALLEL=4 make test +IOGT_TEST_PARALLEL=4 make selenium-test +``` + +More details of the Selenium tests can be found in the [Selenium test README][9]. + [1]: https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment [2]: https://www.unicef.org/innovation/IoGT @@ -199,3 +144,4 @@ Follow instructions [here][10] [8]: ./docs/troubleshooting.md [9]: ./selenium_tests/README.md [10]: ./docs/dependency-management.md +[11]: ./docs/elasticsearch.md diff --git a/docker-compose-legacy-db.yml b/docker-compose-legacy-db.yml index f37ada04b..b2c2e85b6 100644 --- a/docker-compose-legacy-db.yml +++ b/docker-compose-legacy-db.yml @@ -2,7 +2,7 @@ version: '3.5' services: legacy-db: - image: postgres:alpine + image: postgres:9.6-alpine restart: unless-stopped environment: POSTGRES_PASSWORD: iogt diff --git a/docker-compose.yml b/docker-compose.yml index b2a72e272..5feeea4e0 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,70 +1,14 @@ -version: '3.5' +version: '3' services: - django: build: context: ./ dockerfile: Dockerfile.dev environment: - DB_NAME: postgres - DB_USER: postgres - DB_PASSWORD: iogt - DB_HOST: database - DB_PORT: '5432' - DJANGO_SETTINGS_MODULE: iogt.settings.docker_compose_dev - COMMIT_HASH: asdfghjkl - WAGTAILTRANSFER_SECRET_KEY: wagtailtransfer-secret-key - WAGTAILTRANSFER_SOURCE_NAME: iogt_global - WAGTAILTRANSFER_SOURCE_BASE_URL: https://example.com/wagtail-transfer/ - WAGTAILTRANSFER_SOURCE_SECRET_KEY: wagtailtransfer-source-secret-key - BASE_URL: 'http://localhost:8000' - VAPID_PUBLIC_KEY: '' - VAPID_PRIVATE_KEY: '' - VAPID_ADMIN_EMAIL: '' - COMMENTS_COMMUNITY_MODERATION: 'enable' - COMMENT_MODERATION_CLASS: 'comments.clients.AlwaysApproveModerator' - BLACKLISTED_WORDS: '' - SUPERSET_BASE_URL: '' - SUPERSET_DATABASE_NAME: '' - SUPERSET_USERNAME: '' - SUPERSET_PASSWORD: '' - PUSH_NOTIFICATION: 'enable' - JQUERY: 'enable' - MATOMO_TRACKING: 'enable' - MATOMO_SERVER_URL: '' - MATOMO_SITE_ID: '' - IMAGE_SIZE_PRESET: '' + DJANGO_SETTINGS_MODULE: iogt.settings.dev command: python manage.py runserver 0.0.0.0:8000 volumes: - ./:/app/ ports: - "8000:8000" - depends_on: - - elasticsearch - - database - - elasticsearch: - image: 'docker.elastic.co/elasticsearch/elasticsearch:7.12.1' - environment: - - discovery.type=single-node - volumes: - - es-data01:/usr/share/elasticsearch/data - ports: - - "9200:9200" - - database: - image: postgres:alpine - restart: unless-stopped - environment: - POSTGRES_PASSWORD: iogt - volumes: - - pg-data01:/var/lib/postgresql/data - - ./initdb.d:/docker-entrypoint-initdb.d - ports: - - 5432:5432 - - -volumes: - es-data01: - pg-data01: diff --git a/docs/elasticsearch.md b/docs/elasticsearch.md new file mode 100644 index 000000000..c0c7288b0 --- /dev/null +++ b/docs/elasticsearch.md @@ -0,0 +1,64 @@ +# Overview + +Wagtail supports the use of Elasticsearch as a search backend, to speed up searches and remove the burden of searching from the relational database. + +# Setup + +The outline of the setup process is as follows. + +1. Create an Elasticsearch cluster +1. Configure Wagtail to use Elasticsearch +1. Create the search index + +## Create Elasticsearch cluster + +Create the file `docker-compose.override.yml`. Add the following to the file. + +```yaml +version: '3' +services: + search: + image: docker.elastic.co/elasticsearch/elasticsearch:7.12.1 + environment: + - discovery.type=single-node + volumes: + - search:/usr/share/elasticsearch/data + ports: + - "9200:9200" +volumes: + search: +``` + +If you already have a `docker-compose.override.yml`, you will need to merge the snippet above into it, in which case, you will probably not need the first two lines. + +## Configure Wagtail to use Elasticsearch + +Create the file `iogt/settings/local.py`. Add the following to the file. + +```python +WAGTAILSEARCH_BACKENDS = { + 'default': { + 'BACKEND': 'wagtail.search.backends.elasticsearch7', + 'URLS': ['http://search:9200'], + 'INDEX': 'iogt', + 'TIMEOUT': 5, + 'OPTIONS': {}, + 'INDEX_SETTINGS': {}, + 'AUTO_UPDATE': True, + 'ATOMIC_REBUILD': True + } +} +``` + +More information about the available configuration options can be found in the [search backends] section of the Wagtail documentation. + +## Create the search index + +To create and subsequently update the ElasticSearch index. + +```sh +docker compose run --rm django python manage.py update_index +``` + + +[search backends]: https://docs.wagtail.org/en/v2.15.6/topics/search/backends.html diff --git a/docs/envvars.md b/docs/envvars.md new file mode 100644 index 000000000..3f4eb23e7 --- /dev/null +++ b/docs/envvars.md @@ -0,0 +1,46 @@ +# Overview + +The IoGT app supports environment variables as a way of changing some frequently used configuration options. + +# Use with Docker + +Environment variables can be defined in `docker-compose.yml`, `docker-compose.override.yml` or any custom Docker Compose configuration file. Environment variables are specified for each service, for example. + +```yaml +services: + django: + environment: + DB_HOST: db +``` + +# List of environment variables + +```yaml +DB_NAME: postgres +DB_USER: postgres +DB_PASSWORD: iogt +DB_HOST: db +DB_PORT: '5432' +COMMIT_HASH: asdfghjkl +WAGTAILTRANSFER_SECRET_KEY: wagtailtransfer-secret-key +WAGTAILTRANSFER_SOURCE_NAME: iogt_global +WAGTAILTRANSFER_SOURCE_BASE_URL: https://example.com/wagtail-transfer/ +WAGTAILTRANSFER_SOURCE_SECRET_KEY: wagtailtransfer-source-secret-key +VAPID_PUBLIC_KEY: '' +VAPID_PRIVATE_KEY: '' +VAPID_ADMIN_EMAIL: '' +COMMENTS_COMMUNITY_MODERATION: enable +COMMENT_MODERATION_CLASS: comments.clients.AlwaysApproveModerator +BLACKLISTED_WORDS: '' +SUPERSET_BASE_URL: '' +SUPERSET_DATABASE_NAME: '' +SUPERSET_USERNAME: '' +SUPERSET_PASSWORD: '' +PUSH_NOTIFICATION: enable +JQUERY: enable +MATOMO_TRACKING: enable +MATOMO_URL: '' +MATOMO_SITE_ID: '' +IMAGE_SIZE_PRESET: '' +``` + diff --git a/docs/postgresql.md b/docs/postgresql.md new file mode 100644 index 000000000..2fbd4dc5d --- /dev/null +++ b/docs/postgresql.md @@ -0,0 +1,67 @@ +# Overview + +By default, the development configuration uses SQLite as the relational database for the app. A more realistic production configuration might use a high performance relational database lilke PostgreSQL, and it is possible to approximate this kind of setup in development, using Docker. + +# Setup + +In outline: + +1. Create a PostgreSQL database server +1. Configure Wagtail to use PostgreSQL +1. Apply database schema migrations + +## Create a PostgreSQL database server + +Create a file called `docker-compose.override.yaml` and add the following configuration into it. + +```yaml +version: "3" +services: + db: + image: postgres:14-alpine + restart: unless-stopped + environment: + POSTGRES_PASSWORD: iogt + volumes: + - db:/var/lib/postgresql/data + - ./initdb.d:/docker-entrypoint-initdb.d + ports: + - 5432:5432 +volumes: + db: +``` + +Start the service with `docker compose up -d db`. + +## Configure Wagtail to use PostgreSQL + +Create a file called `iogt/settings/local.py` and add the following configuration to it. + +```python +from os import getenv + +DATABASES = { + "default": { + "ENGINE": "django.db.backends.postgresql", + "NAME": getenv("DB_NAME", "postgres"), + "USER": getenv("DB_USER", "postgres"), + "PASSWORD": getenv("DB_PASSWORD", "iogt"), + "HOST": getenv("DB_HOST", "db"), + "PORT": getenv("DB_PORT", "5432"), + } +} +``` + +## Apply database schema migrations + +Create the app database. + +```sh +docker compose run --rm django manage.py migrate +``` + +Start the app. + +```sh +docker compose up -d django +``` diff --git a/iogt/settings/dev.py b/iogt/settings/dev.py index 3900fc5cf..685f81a54 100644 --- a/iogt/settings/dev.py +++ b/iogt/settings/dev.py @@ -1,15 +1,10 @@ from .base import * -# SECURITY WARNING: don't run with debug turned on in production! +BASE_URL = 'http://localhost:8000' DEBUG = True DEBUG_TOOLBAR_ENABLE = False - -# SECURITY WARNING: keep the secret key used in production secret! -SECRET_KEY = 'jjfayj6d=90@@@(rop$98ryt36vuyf3!chtneyoku3_f)*z^h_' - -# SECURITY WARNING: define the correct hosts in production! +SECRET_KEY = '!#secret_key_for_development_only#!' ALLOWED_HOSTS = ['*'] - EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend' if DEBUG and DEBUG_TOOLBAR_ENABLE: diff --git a/iogt/settings/docker_compose_dev.py b/iogt/settings/docker_compose_dev.py deleted file mode 100644 index 4f099199f..000000000 --- a/iogt/settings/docker_compose_dev.py +++ /dev/null @@ -1,28 +0,0 @@ -import os -from .dev import * - -WAGTAILSEARCH_BACKENDS = { - 'default': { - 'BACKEND': 'wagtail.search.backends.elasticsearch7', - 'URLS': ['http://elasticsearch:9200'], - 'INDEX': 'iogt', - 'TIMEOUT': 5, - 'OPTIONS': {}, - 'INDEX_SETTINGS': {}, - 'AUTO_UPDATE': True, - 'ATOMIC_REBUILD': True - } -} - - -# Uncomment if you want to run Postgres -# DATABASES = { -# 'default': { -# 'ENGINE': 'django.db.backends.postgresql', -# 'NAME': os.environ.get('DB_NAME'), -# 'USER': os.environ.get('DB_USER'), -# 'PASSWORD': os.environ.get('DB_PASSWORD'), -# 'HOST': os.environ.get('DB_HOST'), -# 'PORT': os.environ.get('DB_PORT'), -# } -# }