Fix Swagger documentation and review gulpfile #555

tschaffter · 2019-09-25T00:07:16Z

The initial issue is reported in #535.

So far we only had the swagger documentation served when running the portal in development environment. The documentation is served on <portal_url>/api-docs.

In production environment, accessing <portal_url>/api-docs redirects to Swagger pet store example. The issue is that in production environment, comments (and so the swagger documentation) is stripped from the file as part of the build process to minimize the size of the app. Individual files where Swagger was configured to look at also no longer exist.

The solution tested in this PR is to build the swagger document swagger.json and then have the server reading this document when the server starts. The solution developed should support test, development and production environments.

…ild. Currently using hardcoded path that works only on my computer (WIP) (#535)

…ource first)

tschaffter · 2019-09-26T04:23:28Z

A default gulp task could be the following:

gulp.series(
    gulp.parallel('lint', 'test'), 'build', gulp.parallel('watch', 'start')
)

tschaffter · 2019-09-26T04:26:24Z

Review the task build

tschaffter · 2019-09-26T16:50:02Z

Notes on Environment variables (config)

Now

Before deploying the portal in development or production environment, we source a configuration file that includes environment variables. Each configuration file such as envvars-dev and envvars-prod include all the env vars, with a few different values depending on the environment.

Enhancement

The idea is to take inspiration from the approach used for the portal seeds and apply the same approach to configuration. Another motivation is to integrate the reading of the configuration as part of the deployment of the portal without relying any more on the user to run source <file> first.

We would have a default.env file that contains all the env vars required for the portal to run. Depending on the environment run (development, test, production), a second config file is provided that includes only the few env vars that must override the values defined in default.env. These second files would be development.env, test.env and production.env.

These config files need to be used at two different places:

In gulpfile.babel.js
In docker-compose.yml, where several env files can be provided, for example:

env_file:
  - ./common.env
  - ./apps/web.env
  - /opt/secrets.env

Edit: Using env_file is actually not possible as it should be possible to deploy the app using docker-compose up without depending on configuration files such as when we deploy with CodeDeploy, where all the config parameters are provided directly as env vars. Instead, I have written a gulp task to run docker-compose locally. The task does 1) create env vars from the local config file and 2) runs docker-compose up. The approach of loading env vars in gulp task has an extra benefit that is to not polute the env vars of the host.

…ed properly

tschaffter · 2019-09-26T21:16:39Z

Notes on SSL certificates

Now

The SSL certificates (certs/server.cert and server.key) are set as environment variables when sourcing envvars-dev, for example. As we are now reading env vars in gulpfile.babel.js, the content of the SSL certificates is now longer properly done (cat command is not executed).

Solution

Gulp tasks need to be added to load the SSL certificates and assign their content to environment variables. The server codebase must be changed so that it always read the SSL certification from the environment variables.

Done

tschaffter · 2019-09-27T14:15:28Z

Here is the current error encountered by Travis:

[04:27:57] Error: ENOENT: no such file or directory, open 'config/certs/server.cert'

  at Object.openSync (fs.js:443:3)

  at Object.readFileSync (fs.js:343:35)

  at /home/travis/build/Sage-Bionetworks/PHCCollaborationPortal/gulpfile.babel.js:208:41

  at taskWrapper (/home/travis/build/Sage-Bionetworks/PHCCollaborationPortal/node_modules/undertaker/lib/set-task.js:13:15)

  at bound (domain.js:402:14)

  at runBound (domain.js:415:12)

  at asyncRunner (/home/travis/build/Sage-Bionetworks/PHCCollaborationPortal/node_modules/async-done/index.js:55:18)

  at process._tickCallback (internal/process/next_tick.js:61:11)

Solved

tschaffter · 2019-09-28T00:46:16Z

Commenting out Swagger in server/routes.js while working on gulp tasks to run the server.

tschaffter · 2019-09-28T02:42:06Z

Notes on loading of config and SSL certificate/key

Config files in config/*.env are loaded only if they exist
- These files likely exist in the local environment
- These files does not exist on Travis or when deployed using CodeDeploy (passed as env vars)
SSL certificate and private key are loaded only if they exist
- These files likely exist in the local environment
- These files does not exist on Travis or when deployed using CodeDeploy (passed as env vars)

Moreover, here are the current commands to build and start the portal in production environment immediately after having cloned the repos (work in progress).

npm run init:config
npm run init:ssl:server
npm run build
gulp serve:dist

Ultimately I would like to have a command that performs all these tasks, e.g. npm run quickstart.

tschaffter · 2019-09-28T03:37:14Z

Working on a gulp task to start the server in development environment. There is an issue:

[20:33:39] Starting 'start:client'...
ℹ ｢wds｣: Project is running at http://localhost:443/

It should be https://localhost:443/ (not http)

This issue should be resolved with using new WebpackDevServer instead of the command webpack-dev-server.

…nvironment (need to cleanup webpack.server.js)

tschaffter · 2019-09-28T03:52:55Z

Cleanup webpack.server.js and its configuration in webpack.make.js.

tschaffter · 2019-09-28T05:35:56Z

In webpack.make.js, restore TerserPlugin with sourceMap: true.

…e more popular and updated ts-loader

tschaffter · 2019-09-30T04:24:39Z

Notes on MongoDB

I have updated the docker image phccp-mongo a week ago. Mongo now shows a deprecation warning for the following arguments:

phccp-mongo    | 2019-09-30T04:19:26.641+0000 W  CONTROL  [main] Option: sslMode is deprecated. Please use tlsMode instead.
phccp-mongo    | 2019-09-30T04:19:26.643+0000 W  CONTROL  [main] Option: sslPEMKeyFile is deprecated. Please use tlsCertificateKeyFile instead.

Address deprecation warnings (wait first for the docker-compose to work with gulp)

tschaffter · 2019-09-30T04:28:39Z

Notes on Docker and populating DB

This error is reported by the docker container phccp. activities involves making call to phccp-prov, which does not seems to be ready at the time of the population.

finished populating webapps
phccp          | finished populating starred messages
phccp-mongo    | 2019-09-30T04:19:30.426+0000 I  INDEX    [conn3] index build: done building index _id_ on ns phccp.messages
phccp-mongo    | 2019-09-30T04:19:30.428+0000 I  STORAGE  [conn5] createCollection: phccp.threads with generated UUID: 7dbb4ffc-24bb-4c88-992a-43d1bc187adb and options: {}
phccp-mongo    | 2019-09-30T04:19:30.442+0000 I  INDEX    [conn5] index build: done building index _id_ on ns phccp.threads
phccp          | finished populating messages
phccp          | finished populating articles
phccp          | finished populating action permissions
phccp          | finished populating data catalogs
phccp          | finished populating entity notifications
phccp          | finished populating message notifications
phccp          | finished populating reports
phccp          | finished populating states
phccp          | finished populating tools
phccp          | finished populating entity permissions
phccp          | finished populating threads
phccp          | finished populating users
phccp          | error populating activities { RequestError: Error: connect ECONNREFUSED 172.21.0.4:8080
phccp          |     at new RequestError (/usr/src/app/node_modules/request-promise-core/lib/errors.js:14:15)
phccp          |     at Request.plumbing.callback (/usr/src/app/node_modules/request-promise-core/lib/plumbing.js:87:29)
phccp          |     at Request.RP$callback [as _callback] (/usr/src/app/node_modules/request-promise-core/lib/plumbing.js:46:31)
phccp          |     at self.callback (/usr/src/app/node_modules/request/request.js:185:22)
phccp          |     at Request.emit (events.js:189:13)
phccp          |     at Request.onRequestError (/usr/src/app/node_modules/request/request.js:881:8)
phccp          |     at ClientRequest.emit (events.js:189:13)
phccp          |     at Socket.socketErrorListener (_http_client.js:392:9)
phccp          |     at Socket.emit (events.js:189:13)
phccp          |     at emitErrorNT (internal/streams/destroy.js:82:8)
phccp          |     at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)
phccp          |     at process._tickCallback (internal/process/next_tick.js:63:19)
phccp          |   name: 'RequestError',
phccp          |   message: 'Error: connect ECONNREFUSED 172.21.0.4:8080',
phccp          |   cause:
phccp          |    { Error: connect ECONNREFUSED 172.21.0.4:8080
phccp          |        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1097:14)
phccp          |      errno: 'ECONNREFUSED',
phccp          |      code: 'ECONNREFUSED',
phccp          |      syscall: 'connect',
phccp          |      address: '172.21.0.4',
phccp          |      port: 8080 },
phccp          |   error:
phccp          |    { Error: connect ECONNREFUSED 172.21.0.4:8080
phccp          |        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1097:14)
phccp          |      errno: 'ECONNREFUSED',
phccp          |      code: 'ECONNREFUSED',
phccp          |      syscall: 'connect',
phccp          |      address: '172.21.0.4',
phccp          |      port: 8080 },
phccp          |   options:
phccp          |    { method: 'POST',
phccp          |      uri: 'http://phccp-prov:8080/rest/v1/activities/batch',
phccp          |      body: [ [Object], [Object] ],
phccp          |      headers: { 'User-Agent': 'Request-Promise' },
phccp          |      json: true,
phccp          |      callback: [Function: RP$callback],
phccp          |      transform: undefined,
phccp          |      simple: true,
phccp          |      resolveWithFullResponse: false,
phccp          |      transform2xxOnly: false },
phccp          |   response: undefined }
phccp          | Express server listening on 443, in production mode

phccp-prov is only ready later:

phccp-prov     |  * Serving Flask app "synprov.config" (lazy loading)
phccp-prov     |  * Environment: production
phccp-prov     |    WARNING: This is a development server. Do not use it in a production deployment.
phccp-prov     |    Use a production WSGI server instead.
phccp-prov     |  * Debug mode: off
phccp-prov     | INFO:werkzeug: * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)

Fix this issue

tschaffter · 2019-09-30T04:29:42Z

Notes on Docker and missing app.html

Running sudo gulp docker:server:dist shows the following error:

phccp          | Error: ENOENT: no such file or directory, stat '/usr/src/app/client/app.html'

Fix this issue

tschaffter added 4 commits September 24, 2019 16:58

Build swagger.json as part of webpack.make.js when running npm run bu…

a534dc0

…ild. Currently using hardcoded path that works only on my computer (WIP) (#535)

Fix gulp default task

ce8f372

WIP Define gulp tasks to set env vars from file (remove the need to s…

10b4e0d

…ource first)

WIP Loading envvars using dotenv instead of gulp-env

7697ad6

Finalize tasks env:prod and env:test

0a94e15

tschaffter changed the title ~~Fix Swagger documentation in production environment~~ Fix Swagger documentation and review gulpfile Sep 26, 2019

tschaffter added 5 commits September 26, 2019 09:57

Moving to more flexible env vars files

7b4ce16

Review build process up to build:images

521b959

Untrack config files

b4e59f1

Complete review of gulp task build (webpack compile review not covered)

877480d

WIP on start:server:prod, crash because SSL certificates are not load…

a4ff77d

…ed properly

tschaffter added 10 commits September 26, 2019 14:39

Add env:ssl task to gulp

c33116b

Distribution server can be started with gulp serve:dist

8e55d0d

Complete review of the gulp task serve:dist

f667685

Remove font-awesome, fix gulp task lint:server and lint:server:test

e6fc8e3

Add gulp task lint:client:ts

62a5a0d

Add lint:client:html

ff57b70

Complete gulp task lint

9ccd6eb

Add gulp task test:server

2bfb5c5

Add gulp task test

fc13cf6

Review npm cover task, remove references to coverage in gulp file

69fa3fd

tschaffter added 4 commits September 27, 2019 16:48

Add gulp task init:ssl:server, init:ssl:mongodb, init:ssl and init

2b4c9ac

Add gulp tasks init:config:dev, init:config:prod, init:config

20ecca9

Cleanup import in gul pfile

cc41c64

Update the config files

22fabe8

Portal can be started with gulp and gulp serve:dist, work in progress

5737e62

The command sudo gulp serve:dev now works to star the portal in dev e…

2f4d43f

…nvironment (need to cleanup webpack.server.js)

Minor update

0dc7bc3

tschaffter added 10 commits September 28, 2019 09:32

Review of webpack.make.js in progress (done up to devtool)

b2d5b90

Review of babel-loader completed

a13d594

Review asset and html loader, replace awesome-typescript-loader by th…

a03fced

…e more popular and updated ts-loader

Work on webpack.make.js, next is to take care of swagger compilate

b92d4d2

Remove old config files

d553b7a

Reviewed css injection, add comment to webpack.make.js

ea3d291

Remove webpack:dev from serve:dev

c5efe73

Comment out copy:fonts which is not used for now

750dd87

Remove reference to localEnv

1e49dcc

Add a few more comments to webpack.make.js

aa4b5d5

Add prototype of gulp task to run docker-compose

6097a0e

tschaffter mentioned this pull request Oct 18, 2019

/api-docs redirects to Swagger demo Pet Store when to portal is run in production environment #535

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Swagger documentation and review gulpfile #555

Fix Swagger documentation and review gulpfile #555

tschaffter commented Sep 25, 2019 •

edited

Loading

tschaffter commented Sep 26, 2019

tschaffter commented Sep 26, 2019 •

edited

Loading

tschaffter commented Sep 26, 2019 •

edited

Loading

tschaffter commented Sep 26, 2019 •

edited

Loading

tschaffter commented Sep 27, 2019

tschaffter commented Sep 28, 2019

tschaffter commented Sep 28, 2019

tschaffter commented Sep 28, 2019 •

edited

Loading

tschaffter commented Sep 28, 2019 •

edited

Loading

tschaffter commented Sep 28, 2019

tschaffter commented Sep 30, 2019

tschaffter commented Sep 30, 2019

tschaffter commented Sep 30, 2019

Fix Swagger documentation and review gulpfile #555

Are you sure you want to change the base?

Fix Swagger documentation and review gulpfile #555

Conversation

tschaffter commented Sep 25, 2019 • edited Loading

tschaffter commented Sep 26, 2019

tschaffter commented Sep 26, 2019 • edited Loading

tschaffter commented Sep 26, 2019 • edited Loading

Notes on Environment variables (config)

Now

Enhancement

tschaffter commented Sep 26, 2019 • edited Loading

Notes on SSL certificates

Now

Solution

tschaffter commented Sep 27, 2019

tschaffter commented Sep 28, 2019

tschaffter commented Sep 28, 2019

Notes on loading of config and SSL certificate/key

tschaffter commented Sep 28, 2019 • edited Loading

tschaffter commented Sep 28, 2019 • edited Loading

tschaffter commented Sep 28, 2019

tschaffter commented Sep 30, 2019

Notes on MongoDB

tschaffter commented Sep 30, 2019

Notes on Docker and populating DB

tschaffter commented Sep 30, 2019

Notes on Docker and missing app.html

tschaffter commented Sep 25, 2019 •

edited

Loading

tschaffter commented Sep 26, 2019 •

edited

Loading

tschaffter commented Sep 26, 2019 •

edited

Loading

tschaffter commented Sep 26, 2019 •

edited

Loading

tschaffter commented Sep 28, 2019 •

edited

Loading

tschaffter commented Sep 28, 2019 •

edited

Loading