Use docker for development (catima#461)

* Added docker-compose, dockerfile & configuration files
Updated Rails configuration & files for docker
Updated GitHub Actions to use docker psql envars

* Added cron & yarn in dockerfile

* Added supervisor configurations for app & worker
Added postgresql-client12 to dockerfile
Replaced postgresql to postgis docker image
Fixed nginx configuration

* Remove unicorn container
Lunch puma from app container
User db:prepare in setup script
Run setup script everytime app container start

* Add swag:run to app entrypoint
Remove POSTGRES_DB envar
Server logfile to stdout

* Renamed worker to worker-dev
Added catima-redis dependency to catima-app container in docker-composer

* Add mailhog container
Update mail settings to use mailhog when running docker
Update readme for dev with docker

* Typo

* Update readme with docker named volumes

* Fix missing DOCKER_RUNNING envar

* Uncomment some envar (rollback)

* Fix entrypoint comment

* User a separate dockerfile for each container (app, worker)

.github/workflows/api.yml

@@ -10,9 +10,9 @@ jobs:
 
     env:
       CI: gha
-      PG_DATABASE: postgres
-      PG_USER: postgres
-      PG_PASSWORD: postgres
+      POSTGRES_DB: postgres
+      POSTGRES_USER: postgres
+      POSTGRES_PASSWORD: postgres
       RAILS_ENV: test
 
     services:

.github/workflows/ci.yml

@@ -10,9 +10,9 @@ jobs:
 
     env:
       CI: gha
-      PG_DATABASE: postgres
-      PG_USER: postgres
-      PG_PASSWORD: postgres
+      POSTGRES_DB: postgres
+      POSTGRES_USER: postgres
+      POSTGRES_PASSWORD: postgres
       RAILS_ENV: test
 
     services:

.gitignore

@@ -1,3 +1,8 @@
+# Docker
+docker/.env
+supervisord.log
+supervisord.pid
+
 # Ignore bundler config.
 /.bundle
 

README.md

@@ -23,7 +23,9 @@ This README describes the purpose of this repository and how to set up a develop
 * End-user documentation is in [catima/userdoc](https://github.com/catima/userdoc)
 * Development documentation is in [catima/devdoc](https://github.com/catima/devdoc)
 
-## Prerequisites
+## Getting started locally
+
+### Prerequisites
 
 This project requires:
 
@@ -35,8 +37,6 @@ This project requires:
 * Sodium must be installed (`brew install libsodium`, ...)
 * Chrome (for testing with Selenium)
 
-## Getting started
-
 ### bin/setup
 
 Run the `bin/setup` script. This script will:
@@ -53,6 +53,60 @@ Run the `bin/setup` script. This script will:
 2. Install [foreman](https://github.com/ddollar/foreman) with `gem install foreman`
 3. Run `foreman start -f Procfile.dev` to start the Rails app.
 
+## Getting started with docker
+
+### Prerequisites
+
+A working [Docker](https://docs.docker.com/engine/install/) installation is mandatory.
+
+### Docker environment file
+
+Please make sure to copy & rename the **example.env** file to **.env**.
+
+``cp docker/example.env docker/.env``
+
+You can replace the values if needed, but the default ones should work.
+
+### Edit hosts file
+
+Edit hosts file to point **catima.lan** to your docker host.
+
+### Environment installation & configuration
+
+Run the following docker command from the project root directory.
+
+Build & run all the containers for this project.
+
+``docker-compose up`` (add -d if you want to run in the background and silence the logs)
+
+Now you just have to wait for all containers to be created and ready to accept connections (Puma should be started and listening). The setup script will configure the application automatically.
+
+Data for the redis, and postgres services are persisted using docker named volumes. You can see what volumes are currently present with:
+
+``docker volume ls``
+
+If you want to remove a volume (e.g. to start with a fresh database), you can do:
+
+``docker volume rm volume_name``
+
+### Frontend
+
+To access the main application please use the following link.
+
+[http://catima.lan:3000](http://catima.lan:3000)
+
++ [email protected] / admin123
+
+### MailHog
+
+To access mails please use the following link.
+
+[http://catima.lan:8028](http://catima.lan:8028)
+
+Or to get the messages in JSON format.
+
+[http://catima.lan:8028/api/v2/messages](http://catima.lan:8028/api/v2/messages)
+
 ## Tests & API specs
 
 * To run the full suite, run `rails test`

app/mailers/application_mailer.rb

@@ -4,7 +4,7 @@ class ApplicationMailer < ActionMailer::Base
 
   private
 
-  helper_method\
+  helper_method \
   def app_host
     URI(root_url).host
   end

app/views/invitations_mailer/group.html.erb

@@ -4,7 +4,7 @@
   <%= t('.invited_to_access_html',
         invited_by: @invited_by.email,
         group: @group.name,
-        baseurl: ENV['DOMAIN']) %>
+        baseurl: ENV['DOMAIN'].present? ? ENV['DOMAIN'] : 'localhost:3000') %>
 </p>
 
 <%= render('account_information') %>

app/workers/export_worker.rb

@@ -4,7 +4,7 @@ class ExportWorker
   include Sidekiq::Worker
 
   def perform(export_id, locale)
-    dir = Rails.env.development? ? Rails.root.join('tmp', 'exports') : Dir.mktmpdir(SecureRandom.hex)
+    dir = Rails.env.development? ? Rails.root.join('tmp', 'exports', Dir.mktmpdir(SecureRandom.hex)) : Dir.mktmpdir(SecureRandom.hex)
     export = find_export(export_id)
 
     case export.category
@@ -123,7 +123,7 @@ def recursively_deflate_directory(input_dir, disk_file_path, zipfile, zipfile_pa
 
   def put_into_archive(disk_file_path, zipfile, zipfile_path)
     zipfile.get_output_stream(zipfile_path) do |f|
-      f.write(File.open(disk_file_path, 'rb').read)
+      f.write(File.binread(disk_file_path))
     end
   end
 end

bin/setup

@@ -2,9 +2,11 @@
 
 def setup(ci)
   within_project_root do
+    # During CI we do not check the ruby version to allows
+    # testing against multiple versions
     test "ruby -v" => ruby_version unless ci
     run  "gem install bundler:2.1.4 --no-document --conservative"
-    run  "bundle install"
+    run  "bundle install"  if File.exist?("Gemfile.lock")
     run  "bin/yarn install" if File.exist?("yarn.lock")
     run  "bundle exec overcommit --sign"
     run  "bundle exec overcommit --install"
@@ -14,9 +16,7 @@ def setup(ci)
     copy "config/domains.example.yml"
     test_local_env_contains_required_keys
     run  "bundle exec rake locales:generate"
-    run  "bundle exec rake db:create#{':all' unless ci}"
-    run  "bundle exec rake db:migrate"
-    run  "bundle exec rake db:seed"
+    run  "bundle exec rake db:prepare"
   end
 end
 

config/database.example.yml

@@ -17,7 +17,11 @@
 development:
   adapter: postgresql
   encoding: unicode
-  database: viim-core_development
+  database: <%= ENV.fetch('POSTGRES_DB', 'catima_development') %>
+  host: <%= ENV.fetch('POSTGRES_HOST',  'localhost') %>
+  username: <%= ENV.fetch('POSTGRES_USER', nil) %>
+  password: <%= ENV.fetch('POSTGRES_PASSWORD', nil) %>
+  port: <%= ENV.fetch('POSTGRES_PORT', 5432) %>
   min_messages: WARNING
   pool: 5
 
@@ -42,10 +46,10 @@ development:
 test:
   adapter: postgresql
   encoding: unicode
-  database: <%= ENV.fetch('PG_DATABASE', 'viim-core_test') %>
-  host:     <%= ENV.fetch('PG_HOST',  'localhost') %>
-  username: <%= ENV.fetch('PG_USER', nil) %>
-  password: <%= ENV.fetch('PG_PASSWORD', nil) %>
-  port:     <%= ENV.fetch('PG_PORT', 5432) %>
+  database: <%= ENV.fetch('POSTGRES_DB', 'catima_test') %>
+  host: <%= ENV.fetch('POSTGRES_HOST',  'localhost') %>
+  username: <%= ENV.fetch('POSTGRES_USER', nil) %>
+  password: <%= ENV.fetch('POSTGRES_PASSWORD', nil) %>
+  port: <%= ENV.fetch('POSTGRES_PORT', 5432) %>
   min_messages: WARNING
   pool: 5

config/environments/development.rb

@@ -30,11 +30,26 @@
   # Store uploaded files on the local file system (see config/storage.yml for options)
   config.active_storage.service = :local
 
-  # Ensure mailer works in development.
+  # Ensure mailer works in local development
   config.action_mailer.delivery_method = :letter_opener
+
+  # Ensure mailer works in docker development
+  if ENV['DOCKER_RUNNING'].present?
+    config.action_mailer.delivery_method = :smtp
+    config.action_mailer.perform_deliveries = true
+    config.action_mailer.smtp_settings = {
+      address: ENV.fetch('MAIL_SMTP_ADDRESS', 'catima-mailhog'),
+      port: ENV.fetch('MAIL_SMTP_PORT', '1025')
+    }
+  end
+
   config.action_mailer.raise_delivery_errors = true
-  config.action_mailer.default_url_options = { :host => "localhost:3000" }
-  config.action_mailer.asset_host = "http://localhost:3000"
+
+  config.action_mailer.default_url_options = {
+    host: ENV.fetch('DOMAIN', 'localhost:3000'),
+    protocol: ENV.fetch('PROTOCOL', 'http')
+  }
+  config.action_mailer.asset_host = ENV.fetch('ASSET_HOST', 'http://localhost:3000')
 
   config.action_mailer.perform_caching = false
 
@@ -70,7 +85,17 @@
   # Use an evented file watcher to asynchronously detect changes in source code,
   # routes, locales, etc. This feature depends on the listen gem.
   config.file_watcher = ActiveSupport::EventedFileUpdateChecker
+
+  config.hosts = [
+    IPAddr.new("0.0.0.0/0"), # All IPv4 addresses.
+    IPAddr.new("::/0"), # All IPv6 addresses.
+    "localhost", # The localhost reserved domain.
+    "catima.lan"
+  ]
 end
 
 # For building URLs in API resource links
-Rails.application.routes.default_url_options = { :host => "localhost:3000" }
+Rails.application.routes.default_url_options = {
+  :host => ENV.fetch('DOMAIN', 'localhost:3000'),
+  :protocol => ENV.fetch('PROTOCOL', 'http')
+}

db/structure.sql

@@ -2606,3 +2606,4 @@ INSERT INTO "schema_migrations" (version) VALUES
 ('20220510091724'),
 ('20220519124303');
 
+

docker-compose.yml

@@ -0,0 +1,78 @@
+version: "2.4"
+
+networks:
+  catima-net:
+    driver: bridge
+
+services:
+  catima-app:
+    image: unillett/catima:latest-app
+    build:
+      context: ./
+      dockerfile: ./docker/app.dockerfile
+      target: dev
+    env_file:
+      - ./docker/.env
+    ports:
+      - "3000:3000"
+    depends_on:
+      - catima-postgres
+      - catima-redis
+    volumes:
+      - ./:/var/www/catima
+    networks:
+      - catima-net
+    container_name: catima-app
+  catima-worker:
+    image: unillett/catima:latest-worker
+    build:
+      context: ./
+      dockerfile: ./docker/worker.dockerfile
+      target: dev
+    env_file:
+      - ./docker/.env
+    depends_on:
+      - catima-postgres
+      - catima-redis
+      - catima-app
+    volumes_from:
+      - catima-app
+    networks:
+      - catima-net
+    container_name: catima-worker
+  catima-postgres:
+    image: postgis/postgis:12-3.3
+    env_file:
+      - ./docker/.env
+    ports:
+      - "5401:5432"
+    volumes:
+      - postgres-data:/var/lib/postgresql/data
+    networks:
+      - catima-net
+    container_name: catima-postgres
+  catima-redis:
+    image: "redis:5"
+    env_file:
+      - ./docker/.env
+    expose:
+      - "6379"
+    command: redis-server
+    volumes:
+      - redis-data:/data
+    networks:
+      - catima-net
+    container_name: catima-redis
+  catima-mailhog:
+    image: mailhog/mailhog:latest
+    expose:
+      - "1025"
+    ports:
+      - "8028:8025"
+    networks:
+      - catima-net
+    container_name: catima-mailhog
+
+volumes:
+  redis-data:
+  postgres-data:

docker/app.dockerfile

@@ -0,0 +1,59 @@
+FROM ruby:2.7.5 AS base
+
+ENV DOCKER_RUNNING=true
+
+# Update repositories
+RUN apt-get update
+
+# Add needed packages
+RUN apt-get install -y --no-install-recommends \
+    curl \
+    imagemagick \
+    git \
+    zip \
+    supervisor \
+    cron \
+    lsb-release
+
+# Install Postgresql-client 12
+RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - &&\
+    echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" | tee  /etc/apt/sources.list.d/pgdg.list &&\
+    apt-get update &&\
+    apt-get install -y --no-install-recommends postgresql-client-12
+
+# Install Yarn
+RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - &&\
+    echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list &&\
+    apt-get update &&\
+    apt-get install -y --no-install-recommends yarn
+
+# Install Node 16
+RUN curl -sL https://deb.nodesource.com/setup_16.x | bash - &&\
+    apt-get update &&\
+    apt-get install -y --no-install-recommends nodejs
+
+# Create and set the working directory as /var/www/catima
+WORKDIR /var/www/catima
+
+# Update rubygems, install bundler 2.1.4
+RUN gem update --system &&\
+    gem install bundler:2.1.4 --no-document --conservative
+
+# Copy the Gemfile and Gemfile.lock, and run bundle install
+COPY Gemfile /var/www/catima
+COPY Gemfile.lock /var/www/catima
+RUN bundle install
+
+FROM base as dev
+
+# Replace default crontab
+ADD ./docker/config/crontab /etc/crontab
+
+# Copy supervisor configuration file
+COPY ./docker/config/supervisord-app-dev.conf /etc/supervisor/conf.d/supervisord.conf
+
+# Add the entrypoint script used in development
+COPY ./docker/config/entrypoint-dev.sh /usr/local/bin/entrypoint-dev.sh
+RUN chmod +x /usr/local/bin/entrypoint-dev.sh
+ENTRYPOINT [ "entrypoint-dev.sh" ]
+CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]