11 finish events rewrite (#33)

* Whitespace

* Rewrite 11 but good this time

* chore: Split dev/prod/test dbs

- Make database name consistent;
  rename `hsl`, `hsl_dev` and `hsl_test` dbs to `members_api_db_${NODE_ENV}`
- `postgres` container initializes multiple databases
- Rename npm task `up` to `db_migrate_latest`
- Rename npm task `down` to `db_migrate_rollback`
- Rename npm task `seed` to `db_seed`
- Add npm task `db_migrate_up` and `db_migrate_down`
- Add npm task `db_clear`
- Add utility to clear db records between test runs

* get events tests passing, prep app for heroku

* Update readme

* Apply Duncan's fixes for new test methodology

Co-authored-by: Duncan Beevers <[email protected]>

.github/workflows/test.yml

@@ -20,13 +20,13 @@ jobs:
         node-version: '16'
     -
       name: Build docker images
-      run: docker-compose build
+      run: docker compose build
     -
       name: Install dependencies
       uses: bahmutov/npm-install@v1
     -
       name: Run tests
-      run: docker-compose run -e NODE_ENV=test members_api
+      run: NODE_ENV=test docker compose run --rm members_api
 
     -
       name: Cleanup

Dockerfile

@@ -24,4 +24,4 @@ COPY docker-run.sh /home/app/
 
 RUN npm ci
 
-CMD /home/app/docker-run.sh
+CMD "/home/app/docker-run.sh"

README.md

@@ -11,6 +11,7 @@ See the CONTRIBUTING file for information on helping out, or our [Wiki](https://
 ### Prerequisites
 
 - Docker Compose v2 or higher (run `docker compose version` or `docker-compose version`)
+- If using your own database server vs the builtin Docker, Postgres 15.
 
 ### ARM vs x86 Considerations
 
@@ -26,7 +27,7 @@ docker compose up -d members_api
 
 <!--**Consider following the Docker instructions in the `members_app` repo instead of here, to get a full environment going instead of piecemeal with just the API.**-->
 
-If you need to override some of the default environment variables or ports in from the docker-compose.yml file, you can create a docker-compose.override.yml and add your own environment variables or ports there. This file is automatically used by docker-compose and is ignored by git, so you can safely add it to your local repo.
+If you need to override some of the default environment variables or ports in from the docker-compose.yml file, you can create a docker-compose.override.yml and add your own environment variables or ports there.
 
   > **Warning, this docker image is intended for dev mode** attached to a destroyable Postgres server and npm-install-able project folder. ***Running this in critical/prod environments is strongly discouraged without lots of testing!!!***
 
@@ -50,11 +51,18 @@ Review the `Dockerfile` so you know what's about to be booted. For example, the
 Create the docker container for the api and database:
 `docker compose up`
 
-To access the container's shell prompt:
+To access the container's shell prompt for the following npm commands:
 `docker exec -it members_api /bin/sh`
 
-To create basic db tables from within container shell:
-`npm run up`
+If you **do** have an existing db and want to upgrade its schema to the latest structure:
+`npm run db_migrate_latest`
+
+***OR***
+
+If you **don't** have an existing db and want to create basic sample database structure and entries:
+`npm run db_seed`
+
+Other npm tasks include db_migrate_rollback_all, db_migrate_down, db_migrate_up, and db_clear.
 
 To view this container's website from the docker host machine: `http://localhost:3004`
 
@@ -68,19 +76,38 @@ You can build this container directly with: `docker build -t members_api .`
 You can run this container directly with: `docker run -it members_api /bin/sh`
 You'll then have to manually run commands like `npm install` or `npm run start` (see Dockerfile and docker-compose.yml for various assumptions and env vars we use during normal runtime.)
 
+### Emails/SMTP
+
+[Inbucket](https://inbucket.org) is running on http://localhost:10001 by default in dev mode so that you can receive emails without actually sending anything.
+
+## Database
+
+Under Docker, a single container runs two databases; `members_api_db_development` and `members_api_db_test`.
+The test database `members_api_db_test` is managed by Docker scripts, and is migrated up and down as part of each test run.
+The development database is migrated each time it is booted up.
+Migrations and data seeds against the development database may be run manually with the `./scripts/docker-exec.sh` script.
+
+  - `docker exec members_api npm run db_migrate_up`
+  - `docker exec members_api npm run db_seed`
+
 ## Tests
 
-To run tests or coverage, we use lab:
+To run tests or coverage, we use [lab](https://hapi.dev/module/lab/):
 
   - `npm run test`
   - `npm run coverage`
 
 Via Docker:
 
   - `docker-compose build`
-  - `docker-compose run -e NODE_ENV=test members_api`
+  - `NODE_ENV=test docker compose run --rm members_api`
   - `docker stop members_api_postgres && docker container rm members_api_postgres && docker volume rm members_api_db_data`
 
+  or one specific test only:
+
+  - `docker-compose build`
+  - `NODE_ENV=test docker-compose run --rm members_api npm run test test/routes/events/browse.js`
+
 ## Manual testing/usage
 
 Assuming your docker and/or node are on port 3004, you can run:
@@ -104,6 +131,6 @@ Create an event:
 Delete an event:
   `curl -X DELETE -H "Authorization: Bearer $MY_AUTH_TOKEN" localhost:3004/events/YOUR_EVENT_ID_HERE`
 
-## Emails/SMTP
+## Heroku Deploy
 
-[Inbucket](https://inbucket.org) is running on http://localhost:10001 by default in dev mode so that you can receive emails without actually sending anything.
+- See `app.json`

app.json

@@ -0,0 +1,46 @@
+{
+  "name": "Members API",
+  "description": "API for the Members App",
+  "repository": "https://github.com/heatsynclabs/members_api",
+  "env": {
+    "NODE_ENV": {
+      "description": "Run the app in production/development/test",
+      "value": "development"
+    },
+    "JWT_KEY": {
+      "description": "Encryption key for JSON Web Tokens",
+      "value": "some random key"
+    },
+    "PGSSLMODE": {
+      "description": "How to run Postgres SSL",
+      "value": "no-verify"
+    },
+    "SENDGRID_API_KEY": {
+      "description": "API for sending emails when local methods are unavailable"
+    },
+    "ADMIN_EMAIL": {
+      "description": "Address to send email from",
+      "value": "[email protected]"
+    },
+    "SERVER_URL": {
+      "description": "Base URL of this API to link users to when sending them login emails, no trailing slash"
+    },
+    "DOMAIN_LOCATION": {
+      "description": "Base URL of the UI to redirect users on login, no trailing slash"
+    },
+    "DOMAIN_LOCATION_DEV": {
+      "description": "Base URL of the UI to redirect users in dev mode, no trailing slash"
+    },
+    "DEV_COOKIES": {
+      "description": "1 or 0, to set cookie security weak or strong"
+    },
+    "DEV_SAME_SITE": {
+      "description": "1 or 0, to set cookie same-site protection lax or none"
+    }
+  },
+  "addons": [
+    {
+      "plan": "heroku-postgresql"
+    }
+  ]
+}

config.js

@@ -16,10 +16,9 @@ const env = process.env.NODE_ENV || 'development';
 const port = process.env.NODE_PORT || process.env.PORT || 3004;
 const { forEach, startsWith, lowerCase } = require('lodash');
 const Pack = require('./package.json');
-const knexFile = require('./knexfile');
 
 const jwt = {
-  password: process.env.JWT_KEY,
+  password: process.env.JWT_KEY || 'some random key',
   signOptions: {},
 };
 
@@ -48,7 +47,7 @@ forEach(process.env, (v, k) => {
   }
 });
 
-const connectionOptions = {
+const hapiServerOptions = {
   test: {
     host: '0.0.0.0',
     port,
@@ -118,9 +117,9 @@ const cache = {
 
 module.exports = {
   env,
-  knex: knexFile,
   siteName: process.env.SITE_NAME || 'HeatSync Labs',
-  connection: connectionOptions[env],
+  hapiServerOptions: hapiServerOptions[env],
+  databaseUrl: process.env.DATABASE_URL,
   jwt,
   cookies,
   oauth,

docker-compose.yml

@@ -28,9 +28,9 @@ services:
       - "9229:9229"
     environment:
       NODE_PORT: 3004
-      NODE_ENV: "development"
+      NODE_ENV: "${NODE_ENV:-development}"
       NPMINSTALL: 1
-      DATABASE_URL: "postgres://postgres:postgres@members_api_postgres:5432/hsl_dev"
+      DATABASE_URL: "postgres://members_api_db_${NODE_ENV:-development}:members_api_db_${NODE_ENV:-development}@members_api_postgres:5432/members_api_db_${NODE_ENV:-development}"
       SMTP_HOST: "inbucket"
       SMTP_PORT: "10025"
       SMTP_USER: "user"
@@ -45,14 +45,15 @@ services:
 
   members_api_postgres:
     container_name: members_api_postgres
-    image: postgres:11.2-alpine
+    image: postgres:15.1-alpine
     ports:
       - "5432:5432"
     environment:
+      POSTGRES_MULTIPLE_DATABASES: members_api_db_development,members_api_db_test
       POSTGRES_PASSWORD: postgres
-      POSTGRES_DB: hsl_dev
     volumes:
       - db_data:/var/lib/postgresql/data
+      - ./pg-init-scripts:/docker-entrypoint-initdb.d
     healthcheck:
       test: ["CMD-SHELL", "pg_isready -U postgres"]
       interval: 5s

docker-run.sh

@@ -1,46 +1,31 @@
-# Copyright 2019 Iced Development, LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
 echo "Database URL is $DATABASE_URL"
 echo "NODE_ENV is $NODE_ENV"
 
 echo "Setting PATH from $PATH ..."
 export PATH=$PATH:$(pwd)/node_modules/.bin
 echo "... to $PATH"
 
-if [[ $PORT  ]]
+if [ $PORT  ]
 then
   export NODE_PORT=$PORT
   echo "Setting NODE_PORT=$PORT based on PORT"
 fi
-if [[ $NPMINSTALL == 1 ]]
+
+if [ $NPMINSTALL == 1 ]
 then
   npm install
 fi
 
-if [[ $NODE_ENV == "production" ]]
+if [ $NODE_ENV == "production" ]
 then
   npm run start
-elif [[ $NODE_ENV == "development" ]]
+elif [ $NODE_ENV == "development" ]
 then
-  npm run up
-  npm run seed
+  npm run db_migrate_latest
   npm run develop
-elif [[ $NODE_ENV == "test" ]]
+elif [ $NODE_ENV == "test" ]
 then
-  npm run up
+  npm run db_migrate_rollback_all
+  npm run db_migrate_latest
   npm run test
-else
-  echo "Improper NODE_ENV=$NODE_ENV, stopping"
-fi
+fi

index.js

@@ -30,7 +30,7 @@ const handleErrors = require('./handleErrors');
 const Routes = require('./routes');
 
 // Declare a new instance of hapi
-const server = Hapi.server(config.connection);
+const server = Hapi.server(config.hapiServerOptions);
 
 // Start server function
 async function start() {

knex.js

@@ -12,6 +12,6 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-const config = require('./config');
+const knexFile = require('./knexfile');
 
-module.exports = require('knex')(config.knex);
+module.exports = require('knex')(knexFile);

knexfile.js

@@ -12,18 +12,16 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-const config = {
+const config = require('./config');
+
+module.exports = {
   client: 'postgresql',
-  connection: process.env.DATABASE_URL || 'postgres://postgres@localhost:5432/hsl',
+  connection: config.databaseUrl,
   pool: {
     min: 1,
     max: 7,
   },
   seeds: {
-    directory: './migrations/seed'
-  }
+    directory: './migrations/seed',
+  },
 };
-
-// console.log('knex', config);
-
-module.exports = config;

lib/bread.js

@@ -13,8 +13,8 @@
 // limitations under the License.
 
 const breadfruit = require('breadfruit');
-const config = require('../config');
+const knexFile = require('../knexfile');
 
-const bread = breadfruit(config.knex);
+const bread = breadfruit(knexFile);
 
 module.exports = bread;

migrations/20180217155311_initial.js

@@ -13,26 +13,24 @@
 // limitations under the License.
 
 const fs = require('fs');
-const knex = require('../knex')
 
 const up = fs.readFileSync('./migrations/up/20180217155311_initial.sql', 'utf8');
 const down = fs.readFileSync('./migrations/down/20180217155311_initial.sql', 'utf8');
 
-
-exports.up = async function(knexing, success) {
+exports.up = async function (knex) {
   try {
-    await knex.raw(up)
+    await knex.raw(up);
   } catch (err) {
-    console.log('err', err);
+    console.error('err', err);
     return err;
   }
 };
 
-exports.down = async function(success, error) {
+exports.down = async function (knex) {
   try {
-    await knex.raw(down)
+    await knex.raw(down);
   } catch (err) {
-    console.log('err', err);
+    console.error('err', err);
     return err;
   }
 };