feat: Optionally load gatsby datastore in lambdas (#376)

* feat: optionally load Gatsby datastore into lambdas after deploy

* chore: add logging to determine code path

* fix: dataMetadata.json file wasn't being included in function bundle

* fix: await promise returned from prepareFilesystem

Suspect that the function is completing early because this wasn't happening before

* fix: address 'handler is not a function' err

* fix: move prepareFilesystem check within returned handler

* chore: add logging

* fix: downloadUrl is incorrect

* fix: round 2 on the url

* fix: only copy file that we're interested in for mvp

* fix: create directory to copy file into before streaming

* fix: specify the full filepath

thought this would download the file into the directory, but suspecting that the file needs to exist
before the stream writes

* style: lint fixes

* feat: add pre-warm requests to lambda endpoints

aiming to mitigate the effects of the cold start when downloading the
gatsby datastore from the CDN

* fix: use https instead of fetch for request

* fix: mistake with the func path

* test: not seeing prewarm behaviour, add logs

* style: lint fixes

* refactor: rename e2e test folder

* test: update test command in package.json

* refactor: more organizing

* test: add tests for downloadFile helper method

* style: lint

* chore: add tsconfig.json

small cleanups

* chore: cleanup

* test: add createDatastoreMetadataFile test

* chore(test): rename file

* test: add onSuccess test

add chance.js as dependency for randomized values

* chore: make changes flagged by linter

* style: lint

* fix: explicitly check env var is 'true'

If LOAD_GATSBY_LMDB_DATASTORE_FROM_CDN were set to 'false',
the original conditional would still evaluate to 'true' when not cast as a boolean.

* test: add coverage for onBuild method

* style: lint changes

* style: lint fixes

* refactor: revert reorg of test files

Will do this separately in order to reduce amount of changes in this PR

* refactor: remove README.md from .gitignore

revert change to command to run e2e tests

* fix: check value in case it's set to 'false'

* test: add mutateConfig coverage

* test: revert accidental change to jest config

* test: add test for prepareFilesystem

* feat: add timeout to pre-warm requests

requires a minimum supported node version of 14.17 as
that's when AbortController was introduced globally

* style: lint

* test: update tests after merge

* style: lint

* ci: add gatsby to global install step

the demo project is being built in some tests and requires gatsby to be available

* ci: forgot to add gatsby dependency to ubuntu tests

* test: remove gatsby install

wasn't having desired effect. Try something else

* ci: revert change

* Apply suggestions from code review

Co-authored-by: Matt Kane <[email protected]>

* fix: address some code review comments

* fix: address some code review comments

* fix: address cr comments

move the dataMetadata.json file to the .cache directory rather than public.

* fix: don't generate unique id for datastore every build

reuse the existing ID if a datastore file already exists in the publish directory

* refactor: isEnvSet

* ci: install demo/ deps

Attempting to address missing gatsby error in test builds

* ci: fix silly mistake

* test: increase timeout

the tests currently failing on build are long running ones, and the Github
 workers might need more time than a local machine

* test: use netlify-cli to build project

tests were failing on github workers because the wrong command to build was being used

* ci: add install command back in

* fix: hard code env var check

* style: lint

* fix: address cr comments

add check for dataMetadata.json file in order to use filename there if it exists

* fix: regenerate package-lock to fix conflicts

* fix: regenerate package-lock

* chore: update @netlify/build resolution

package resolution wasn't picking up fix to netlify/esbuild#17

* chore: run npm update on plugin/

* chore: check in working version of plugin lockfile

* chore: lockfile

* chore: add lockfile back in

* chore: update lint rules

increase version of node to 14

* chore: pin eslint-config-node

* test: add axios to invalidate the github test cache

* chore: increase node version

remove axios as well

* refactor: attempt to read metadata file first

this avoids needing to traverse the list of files to reuse the name when possible

* chore(deps): move execa to devDep in plugin

* style: lint

Co-authored-by: Matt Kane <[email protected]>

.eslintrc.js

@@ -1,7 +1,7 @@
 const { overrides } = require('@netlify/eslint-config-node')
 
 module.exports = {
-  extends: '@netlify/eslint-config-node',
+  extends: ['@netlify/eslint-config-node'],
   rules: {
     // For now
     'func-style': 'off',
@@ -37,12 +37,26 @@ module.exports = {
         'import/no-unresolved': 'off',
         'node/no-unpublished-require': 'off',
         'node/no-missing-require': 'off',
+        'node/no-missing-import': 'off',
         'max-lines': 'off',
         complexity: 'off',
         'max-statements': 'off',
         'node/prefer-global/process': 'off',
         'unicorn/filename-case': 'off',
       },
     },
+    {
+      files: ['plugin/test/**/*'],
+      rules: {
+        'node/no-unpublished-import': 'off',
+        'max-lines': 'off',
+        'node/no-unpublished-require': 'off',
+        'node/no-missing-require': 'off',
+        'node/no-missing-import': 'off',
+        '@typescript-eslint/no-var-requires': 'off',
+        'node/global-require': 'off',
+        'node/prefer-global/process': 'off',
+      },
+    },
   ],
 }

demo/package.json

@@ -47,6 +47,6 @@
   },
   "license": "MIT",
   "engines": {
-    "node": ">=14.15.0"
+    "node": ">=14.17.0"
   }
 }

jest.config.js

@@ -0,0 +1,6 @@
+/** @type {import('ts-jest/dist/types').InitialOptionsTsJest} */
+module.exports = {
+  preset: 'ts-jest',
+  verbose: true,
+  testMatch: ['<rootDir>/plugin/test/unit/**/*.spec.ts'],
+}

netlify.toml

@@ -2,4 +2,4 @@
 base="demo/"
 
 [build.environment]
-NODE_VERSION = "12.13.0"
+NODE_VERSION = "14.17.0"