Support federated API deployment

CHANGELOG.md

@@ -2,7 +2,33 @@
 
 All changes that impact users of this module are documented in this file, in the [Common Changelog](https://common-changelog.org) format with some additional specifications defined in the CONTRIBUTING file. This codebase adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## Unreleased
+## Unreleased [minor]
+
+> Development of this release was [supported](https://nlnet.nl/project/TOSDR-OTA/) by the [NGI0 Entrust Fund](https://nlnet.nl/entrust), a fund established by [NLnet](https://nlnet.nl/) with financial support from the European Commission's [Next Generation Internet](https://www.ngi.eu) programme, under the aegis of DG CNECT under grant agreement N°101069594.
+
+### Changed
+
+- **Breaking:** Rename `deploy` playbook to setup the infrastructure and deploy engine to `engine.all`; update your scripts accordingly
+- **Breaking:** Rename variable `ota_config_path` to `ota_engine_config_path`; update your inventory file accordingly
+- **Breaking:** Rename variable `ota_declarations_directory` to `ota_engine_declarations_directory`; update your inventory file accordingly
+- **Breaking:** Rename variable `ota_declarations_branch` to `ota_engine_declarations_branch`; update your inventory file accordingly
+- **Breaking:** Rename variable `ota_snapshots_branch` to `ota_engine_snapshots_branch`; update your inventory file accordingly
+- **Breaking:** Rename variable `ota_versions_branch` to `ota_engine_versions_branch`; update your inventory file accordingly
+
+### Removed
+
+- **Breaking:** Remove tag `setup`; use the application deployment playbook as replacement in your scripts `ansible-playbook opentermsarchive.deployment.engine.application`
+- **Breaking:** Remove tag `update`; use the application deployment playbook as replacement in your scripts `ansible-playbook opentermsarchive.deployment.engine.application`
+- **Breaking:** Remove tag `engine`; use the application deployment playbook as replacement in your scripts `ansible-playbook opentermsarchive.deployment.engine.application`
+- **Breaking:** Remove tag `infra`; use the infrastructure set up playbook as replacement in your scripts `ansible-playbook opentermsarchive.deployment.engine.infrastructure`
+
+### Added
+
+- Add playbook to setup and deploy the Open Terms Archive federated API
+
+### Fixed
+
+- Ensure that the configuration for the port of the collection metadata API is successfully applied to the Nginx configuration
 
 ## 0.0.14 - 2023-10-11
 

README.md

@@ -1,8 +1,8 @@
 # Open Terms Archive Ansible Collection
 
-This repository contains the `opentermsarchive.deployment` Ansible Collection. This ansible collection provides roles and playbooks to set up the infrastructure of and deploy Open Terms Archive.
+This repository contains the `opentermsarchive.deployment` Ansible collection. This Ansible collection provides playbooks to set up the infrastructure of and deploy Open Terms Archive applications.
 
-## Usage
+## Installation
 
 [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) is required to use this collection.
 
@@ -26,77 +26,117 @@ And installed with the command:
 ansible-galaxy collection install -r requirements.yml
 ```
 
-Once installed, the `deploy` playbook can be used using the `ansible-playbook` command-line tool:
+## Usage
+
+Once installed, some playbooks are available to deploy the two main Open Terms Archive applications: [Engine](https://github.com/OpenTermsArchive/engine) and [Federated API](https://github.com/OpenTermsArchive/federated-api).
+
+Each playbook can be executed using the `ansible-playbook` command-line tool:
 
 ```sh
-ansible-playbook opentermsarchive.deployment.deploy
+ansible-playbook opentermsarchive.deployment.<playbook-name>
 ```
 
-See [Ansible Using collections](https://docs.ansible.com/ansible/latest/user_guide/collections_using.html) for more information about ansible collections.
+Refer to the application related sections below for a list of available playbooks.
 
-### Configuration
+_It is possible to check a playbook execution without actually applying changes with `check` and `diff` options:_
 
-Available variables are listed below, along with default values:
+```sh
+ansible-playbook opentermsarchive.deployment.<playbook-name> --check --diff
+```
 
-- `ota_config_path`: Path to the engine config file related to the inventory file. Default: `../config/production.json`.
-- `ota_declarations_branch`: Git branch of the declarations repository to use. Default: `main`.
-- `ota_snapshots_branch`: Git branch of the snapshots repository to use. Default: `main`.
-- `ota_versions_branch`: Git branch of the versions repository to use. Default: `main`.
-- `ota_declarations_directory`: Directory path where the code will be deploy on the server. Default: the value declared in the `name` key in the engine config file.
+> See “[Using collections](https://docs.ansible.com/ansible/latest/user_guide/collections_using.html)” in Ansible’s user guide for more information about Ansible collections.
 
-### Commands
+- - -
 
-- To set up a full [(phoenix)](https://martinfowler.com/bliki/PhoenixServer.html) server:
-```
-ansible-playbook opentermsarchive.deployment.deploy
-```
+### Engine application
 
-Some [tags](https://docs.ansible.com/ansible/latest/user_guide/playbooks_tags.html) are available to refine what will happen, use them with `--tags`:
+Available playbooks for the engine application:
 
-- **Server setup:**
-    - `infra`: to only setup the infrastructure
-    - `engine`: to only setup the `Open Terms Archive` engine (the infrastructure must have been put in place at least once before)
-- **Engine control:**
-    - `setup`: to only setup system dependencies required by the engine (cloning repo, installing engine dependencies, all config files, and so on…)
-    - `start`: to start the engine
-    - `stop`: to stop the engine
-    - `restart`: to restart the engine
-    - `update`: to update the engine (pull code, install dependencies and restart engine)
-    - `update-declarations`: to update services declarations (pull declarations, install dependencies and restart engine)
+| Playbook name | Description | Command example |
+| --- | --- | --- |
+| `engine.infrastructure` | Set up and configure the infrastructure required by the Open Terms Archive engine | `ansible-playbook opentermsarchive.deployment.engine.infrastructure` |
+| `engine.application` | Deploy the Open Terms Archive engine | `ansible-playbook opentermsarchive.deployment.engine.application` |
+| `engine.all` | Set up infrastructure and deploy the Open Terms Archive engine | `ansible-playbook opentermsarchive.deployment.engine.all` |
 
+#### Configuration
 
-#### Refined commands examples
+Available [variables](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html) are listed below, along with default values:
 
-- To setup the infrastructure only:
-```sh
-ansible-playbook opentermsarchive.deployment.deploy --tags infra
-```
+| Variable | Description | Default value |
+| --- | --- | --- |
+| `ota_engine_config_path` | Path to the engine config file related to the inventory file | `../config/production.json` |
+| `ota_engine_declarations_branch` | Git branch of the declarations repository to use | `main` |
+| `ota_engine_snapshots_branch` | Git branch of the snapshots repository to use | `main` |
+| `ota_engine_versions_branch` | Git branch of the versions repository to use | `main` |
+| `ota_engine_declarations_directory` | Path of the directory where the code will be deployed on the server | Value declared in the `name` key in the engine config file |
 
-- To setup the `Open Terms Archive` engine only:
-```sh
-ansible-playbook opentermsarchive.deployment.deploy --tags engine
-```
+These variables can be overriden in the inventory file, for example:
 
-- Check deployment without actually applying changes:
-```sh
-ansible-playbook opentermsarchive.deployment.deploy --check --diff
+```yml
+all:
+  hosts:
+    127.0.0.1:
+      ansible_user: debian
+      ota_engine_config_path: ./engine_config.json
+      ota_engine_declarations_branch: new-feature
 ```
 
-- Update the Open Terms Archive engine only, without applying changes to the infrastructure:
-```sh
-ansible-playbook opentermsarchive.deployment.deploy --tags update
-```
+#### Tags
 
-- Update services declarations only:
-```sh
-ansible-playbook opentermsarchive.deployment.deploy --tags update-declarations
-```
+Available [tags](https://docs.ansible.com/ansible/latest/user_guide/playbooks_tags.html) to refine what will happen, use them with `--tags`:
 
-- Stop the Open Terms Archive engine only:
-```sh
-ansible-playbook opentermsarchive.deployment.deploy --tags stop
+| Tag | Description | Command example |
+| --- | --- | --- |
+| `start` | Start the engine | `ansible-playbook opentermsarchive.deployment.engine.application --tags start` |
+| `stop` | Stop the engine | `ansible-playbook opentermsarchive.deployment.engine.application --tags stop` |
+| `restart` | Restart the engine | `ansible-playbook opentermsarchive.deployment.engine.application --tags restart` |
+| `update-declarations` | Update service declarations (pull declarations, install dependencies, and restart engine) | `ansible-playbook opentermsarchive.deployment.engine.application --tags update-declarations` |
+
+- - -
+
+### Federated API application
+
+Available playbooks for the Federated API application:
+
+| Playbook name | Description | Command example |
+| --- | --- | --- |
+| `federated_api.infrastructure` | Set up and configure the infrastructure required by the Open Terms Archive federated API | `ansible-playbook opentermsarchive.deployment.federated_api.infrastructure` |
+| `federated_api.application` | Deploy the Open Terms Archive federated API | `ansible-playbook opentermsarchive.deployment.federated_api.application` |
+| `federated_api.all` | Set up infrastructure and deploy the Open Terms Archive federated API | `ansible-playbook opentermsarchive.deployment.federated_api.all` |
+
+#### Configuration
+
+Available variables are listed below, along with default values:
+
+| Variable | Description | Default value |
+| --- | --- | --- |
+| `ota_federated_api_repo` | Repository URL of the federated API code | `https://github.com/OpenTermsArchive/federated-api.git` |
+| `ota_federated_api_directory` | Path of the directory where the code will be deployed on the server | `federated-api` |
+| `ota_federated_api_branch` | Git branch of the federated API repository to use | `main` |
+
+These variables can be overridden in the inventory file, for example:
+
+```yml
+all:
+  hosts:
+    127.0.0.1:
+      ansible_user: debian
+      ota_federated_api_repo: https://github.com/OpenTermsArchive/federated-api.git
+      ota_federated_api_branch: new-feature
 ```
 
+#### Tags
+
+Available [tags](https://docs.ansible.com/ansible/latest/user_guide/playbooks_tags.html) to refine what will happen, use them with `--tags`:
+
+| Tag | Description | Command example |
+| --- | --- | --- |
+| `start` | To only start the Federated API | `ansible-playbook opentermsarchive.deployment.federated_api.application --tags start` |
+| `stop` | To only stop the Federated API | `ansible-playbook opentermsarchive.deployment.federated_api.application --tags stop` |
+| `restart` | To only restart the Federated API | `ansible-playbook opentermsarchive.deployment.federated_api.application --tags restart` |
+
+- - -
+
 ## Development
 
 ### Requirements
@@ -139,7 +179,7 @@ vagrant up
 Start by applying changes on the virtual machine:
 
 ```sh
-ansible-playbook ../playbooks/deploy.yml
+ansible-playbook ../playbooks/engine/all.yml
 ```
 
 Connect through SSH to the virtual machine and check that everything works as intended:

playbooks/engine/all.yml

@@ -0,0 +1,6 @@
+---
+- name: Set up infrastructure and deploy the Open Terms Archive engine
+  hosts: all
+
+- import_playbook: infrastructure.yml
+- import_playbook: application.yml

playbooks/engine/application.yml

@@ -0,0 +1,14 @@
+---
+- name: Deploy the Open Terms Archive engine 
+  hosts: all
+
+  tasks:
+    - name: Load the production config
+      ansible.builtin.include_vars:
+        name: app_config
+        file: "{{ inventory_dir }}/{{ ota_engine_config_path | default('../config/production.json') }}"
+      tags: always
+
+    - ansible.builtin.include_role:
+        name: engine
+      tags: always

playbooks/engine/infrastructure.yml

@@ -0,0 +1,26 @@
+---
+- name: Set up infrastructure
+  hosts: all
+  become: true
+
+  tasks:
+    - name: Load OTA engine config
+      ansible.builtin.include_vars:
+        name: app_config
+        file: "{{ inventory_dir }}/{{ ota_engine_config_path | default('../config/production.json') }}"
+      tags: always
+
+    - ansible.builtin.include_role:
+        name: infrastructure/mongo
+      when:
+        - (app_config.recorder.versions.storage.type is defined and app_config.recorder.versions.storage.type == 'mongo') or
+          (app_config.recorder.snapshots.storage.type is defined and app_config.recorder.snapshots.storage.type == 'mongo')
+        # Skip Debian 11 with ARM architecture as it is not currently supported by MongoDB.
+        # See https://www.mongodb.com/docs/manual/installation/#supported-platforms
+        - ansible_distribution != 'Debian' or (ansible_distribution == 'Debian' and ansible_facts['architecture'] != 'aarch64')
+
+  roles:
+    - role: infrastructure/git
+    - role: infrastructure/node
+    - role: infrastructure/chromium
+    - role: infrastructure/nginx

playbooks/federated_api/all.yml

@@ -0,0 +1,6 @@
+---
+- name: Set up infrastructure and deploy the Open Terms Archive federated API
+  hosts: all
+
+- import_playbook: infrastructure.yml
+- import_playbook: application.yml

playbooks/federated_api/application.yml

@@ -0,0 +1,6 @@
+---
+- name: Deploy the Open Terms Archive federated API 
+  hosts: all
+
+  roles:
+    - role: federated-api

playbooks/federated_api/infrastructure.yml

@@ -0,0 +1,9 @@
+---
+- name: Set up infrastructure
+  hosts: all
+  become: true
+
+  roles:
+    - role: infrastructure/git
+    - role: infrastructure/node
+    - role: infrastructure/nginx

roles/engine/defaults/main.yml

@@ -1,4 +1,4 @@
-ota_declarations_directory: "{{ app_config.name }}"
-ota_declarations_branch: main
-ota_snapshots_branch: main
-ota_versions_branch: main
+ota_engine_declarations_directory: "{{ app_config.name }}"
+ota_engine_declarations_branch: main
+ota_engine_snapshots_branch: main
+ota_engine_versions_branch: main

roles/engine/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: Restart NGINX
+  become: true
+  ansible.builtin.service:
+    name: nginx
+    state: restarted

roles/engine/tasks/database.yml

@@ -10,11 +10,6 @@
   # the `before` property of the return value can tell us if the repository has been cloned already or not,
   # see <https://docs.ansible.com/ansible/latest/collections/ansible/builtin/git_module.html#return-values>
   register: existing_repository
-  tags:
-    - restart
-    - start
-    - update
-    - setup
 
 - name: Obtain {{ engine_database_name }} initial data from branch {{ engine_database_branch }} of {{ engine_database_repository }}
   ansible.builtin.git:
@@ -24,44 +19,23 @@
     accept_hostkey: true
     key_file: '/home/{{ ansible_user }}/.ssh/ota-bot-key'
   when: existing_repository.before is defined and not existing_repository.before # if existing_repository.before is null, then the repository is new
-  tags:
-    - setup
 
 - name: Remove existing locks in {{ engine_database_name }}
   ansible.builtin.file:
     path: '{{ engine_database_directory }}/.git/index.lock'
     state: absent
-  tags:
-    - restart
-    - start
-    - update
 
 - name: Get latest data from {{ engine_database_repository }}
   ansible.builtin.command:
     cmd: git fetch origin
     chdir: '{{ engine_database_directory }}'
-  tags:
-    - restart
-    - start
-    - update
-    - setup
 
 - name: Clean {{ engine_database_name }} local copy
   ansible.builtin.command:
     cmd: git reset --hard origin/{{ engine_database_branch }}
     chdir: '{{ engine_database_directory }}'
-  tags:
-    - restart
-    - start
-    - update
-    - setup
 
 - name: Ensure {{ engine_database_name }} is on branch {{ engine_database_branch }}
   ansible.builtin.command:
     cmd: git checkout {{ engine_database_branch }}
     chdir: '{{ engine_database_directory }}'
-  tags:
-    - restart
-    - start
-    - update
-    - setup