electron-forge make node-pre-gyp error

[stefanJi]

electron-forge make --platform=darwin --targets=@electron-forge/maker-dmg

An unhandled error has occurred inside Forge:
Compilation errors in the main process: assets by status 3.48 MiB [cached] 5 assets

ERROR in ./node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js 43:20-42
Module not found: Error: Can't resolve 'mock-aws-s3' in '/Users/xxx/Documents/xxx_repo/yy/node_modules/@mapbox/node-pre-gyp/lib/util'
 @ ./node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js 15:21-62
 @ ./node_modules/sqlite3/lib/sqlite3-binding.js 1:13-44
 @ ./node_modules/sqlite3/lib/sqlite3.js 2:14-45
 @ ./src/main/tt/sqlitedb.ts 11:0-35 26:28-44
 @ ./src/main/tt/sdkwrapper.ts 17:0-44 43:24-38
 @ ./src/main/ipc/MainIpc.ts 4:0-54 36:19-50 39:19-57 42:19-67 45:19-55 48:19-52
 @ ./src/main/index.ts 8:0-40 22:8-20

ERROR in ./node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js 76:14-32
Module not found: Error: Can't resolve 'aws-sdk' in '/Users/xxx/Documents/xxx_repo/yy/node_modules/@mapbox/node-pre-gyp/lib/util'
 @ ./node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js 15:21-62
 @ ./node_modules/sqlite3/lib/sqlite3-binding.js 1:13-44
 @ ./node_modules/sqlite3/lib/sqlite3.js 2:14-45
 @ ./src/main/tt/sqlitedb.ts 11:0-35 26:28-44
 @ ./src/main/tt/sdkwrapper.ts 17:0-44 43:24-38
 @ ./src/main/ipc/MainIpc.ts 4:0-54 36:19-50 39:19-57 42:19-67 45:19-55 48:19-52
 @ ./src/main/index.ts 8:0-40 22:8-20

ERROR in ./node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js 112:15-30
Module not found: Error: Can't resolve 'nock' in '/Users/xxx/Documents/xxx_repo/yy/node_modules/@mapbox/node-pre-gyp/lib/util'
 @ ./node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js 15:21-62
 @ ./node_modules/sqlite3/lib/sqlite3-binding.js 1:13-44
 @ ./node_modules/sqlite3/lib/sqlite3.js 2:14-45
 @ ./src/main/tt/sqlitedb.ts 11:0-35 26:28-44
 @ ./src/main/tt/sdkwrapper.ts 17:0-44 43:24-38
 @ ./src/main/ipc/MainIpc.ts 4:0-54 36:19-50 39:19-57 42:19-67 45:19-55 48:19-52
 @ ./src/main/index.ts 8:0-40 22:8-20

3 errors have detailed information that is not shown.
Use 'stats.errorDetails: true' resp. '--stats-error-details' to show it.

webpack 5.72.0 compiled with 3 errors and 2 warnings in 14421 ms
Error: Compilation errors in the main process: assets by status 3.48 MiB [cached] 5 assets
orphan modules 1.78 MiB [orphan] 137 modules
runtime modules 1.02 KiB 5 modules
javascript modules 2.71 MiB
  cacheable modules 2.71 MiB 329 modules
  + 19 modules
json modules 409 KiB
  modules by path ./node_modules/@byted-jiyangjy/im-web-sdk/dist/esm/ 383 KiB
    ./node_modules/@byted-jiyangjy/im-web-sdk/dist/esm/emoji/res/images/imsdk_emoji_1.json 3.46 KiB [optional] [built] [code generated]
    ./node_modules/@byted-jiyangjy/im-web-sdk/dist/esm/emoji/res/images/imsdk_emoji_10.json 3.31 KiB [optional] [built] [code generated]
    ./node_modules/@byted-jiyangjy/im-web-sdk/dist/esm/emoji/res/images/imsdk_emoji_100.json 3.13 KiB [optional] [built] [code generated]
    + 111 modules
  modules by path ./node_modules/@mapbox/node-pre-gyp/ 24.2 KiB
    ./node_modules/@mapbox/node-pre-gyp/package.json 1.29 KiB [built] [code generated]
    ./node_modules/@mapbox/node-pre-gyp/lib/util/abi_crosswalk.json 22.9 KiB [built] [code generated]
  ./node_modules/axios/package.json 1.92 KiB [built] [code generated]

WARNING in ./node_modules/ws/lib/buffer-util.js 103:21-42
Module not found: Error: Can't resolve 'bufferutil' in '/Users/xxx/Documents/xxx_repo/yy/node_modules/ws/lib'
 @ ./node_modules/ws/lib/receiver.js 12:42-66
 @ ./node_modules/ws/wrapper.mjs 2:0-41 7:0-79
 @ ./src/main/tt/wsclient.ts 3:0-31 27:16-25 49:63-77
 @ ./src/main/tt/sdkwrapper.ts 18:0-46 51:27-43
 @ ./src/main/ipc/MainIpc.ts 4:0-54 36:19-50 39:19-57 42:19-67 45:19-55 48:19-52
 @ ./src/main/index.ts 8:0-40 22:8-20

WARNING in ./node_modules/ws/lib/validation.js 109:22-47
Module not found: Error: Can't resolve 'utf-8-validate' in '/Users/xxx/Documents/xxx_repo/yy/node_modules/ws/lib'
 @ ./node_modules/ws/lib/receiver.js 13:43-66
 @ ./node_modules/ws/wrapper.mjs 2:0-41 7:0-79
 @ ./src/main/tt/wsclient.ts 3:0-31 27:16-25 49:63-77
 @ ./src/main/tt/sdkwrapper.ts 18:0-46 51:27-43
 @ ./src/main/ipc/MainIpc.ts 4:0-54 36:19-50 39:19-57 42:19-67 45:19-55 48:19-52
 @ ./src/main/index.ts 8:0-40 22:8-20

2 warnings have detailed information that is not shown.
Use 'stats.errorDetails: true' resp. '--stats-error-details' to show it.

ERROR in ./node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js 43:20-42
Module not found: Error: Can't resolve 'mock-aws-s3' in '/Users/xxx/Documents/xxx_repo/yy/node_modules/@mapbox/node-pre-gyp/lib/util'
 @ ./node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js 15:21-62
 @ ./node_modules/sqlite3/lib/sqlite3-binding.js 1:13-44
 @ ./node_modules/sqlite3/lib/sqlite3.js 2:14-45
 @ ./src/main/tt/sqlitedb.ts 11:0-35 26:28-44
 @ ./src/main/tt/sdkwrapper.ts 17:0-44 43:24-38
 @ ./src/main/ipc/MainIpc.ts 4:0-54 36:19-50 39:19-57 42:19-67 45:19-55 48:19-52
 @ ./src/main/index.ts 8:0-40 22:8-20

ERROR in ./node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js 76:14-32
Module not found: Error: Can't resolve 'aws-sdk' in '/Users/xxx/Documents/xxx_repo/yy/node_modules/@mapbox/node-pre-gyp/lib/util'
 @ ./node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js 15:21-62
 @ ./node_modules/sqlite3/lib/sqlite3-binding.js 1:13-44
 @ ./node_modules/sqlite3/lib/sqlite3.js 2:14-45
 @ ./src/main/tt/sqlitedb.ts 11:0-35 26:28-44
 @ ./src/main/tt/sdkwrapper.ts 17:0-44 43:24-38
 @ ./src/main/ipc/MainIpc.ts 4:0-54 36:19-50 39:19-57 42:19-67 45:19-55 48:19-52
 @ ./src/main/index.ts 8:0-40 22:8-20

ERROR in ./node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js 112:15-30
Module not found: Error: Can't resolve 'nock' in '/Users/xxx/Documents/xxx_repo/yy/node_modules/@mapbox/node-pre-gyp/lib/util'
 @ ./node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js 15:21-62
 @ ./node_modules/sqlite3/lib/sqlite3-binding.js 1:13-44
 @ ./node_modules/sqlite3/lib/sqlite3.js 2:14-45
 @ ./src/main/tt/sqlitedb.ts 11:0-35 26:28-44
 @ ./src/main/tt/sdkwrapper.ts 17:0-44 43:24-38
 @ ./src/main/ipc/MainIpc.ts 4:0-54 36:19-50 39:19-57 42:19-67 45:19-55 48:19-52
 @ ./src/main/index.ts 8:0-40 22:8-20

3 errors have detailed information that is not shown.
Use 'stats.errorDetails: true' resp. '--stats-error-details' to show it.

webpack 5.72.0 compiled with 3 errors and 2 warnings in 14421 ms
    at cb (/Users/xxx/Documents/xxx_repo/yy/node_modules/@electron-forge/plugin-webpack/src/WebpackPlugin.ts:270:31)
    at finalCallback (/Users/xxx/Documents/xxx_repo/yy/node_modules/webpack/lib/Compiler.js:441:32)
    at /Users/xxx/Documents/xxx_repo/yy/node_modules/webpack/lib/Compiler.js:458:13
    at Hook.eval [as callAsync] (eval at create (/Users/xxx/Documents/xxx_repo/yy/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/Users/xxx/Documents/xxx_repo/yy/node_modules/tapable/lib/Hook.js:18:14)
    at onCompiled (/Users/xxx/Documents/xxx_repo/yy/node_modules/webpack/lib/Compiler.js:456:21)
    at /Users/xxx/Documents/xxx_repo/yy/node_modules/webpack/lib/Compiler.js:1196:17
    at Hook.eval [as callAsync] (eval at create (/Users/xxx/Documents/xxx_repo/yy/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/Users/xxx/Documents/xxx_repo/yy/node_modules/tapable/lib/Hook.js:18:14)
    at /Users/xxx/Documents/xxx_repo/yy/node_modules/webpack/lib/Compiler.js:1192:33
    at finalCallback (/Users/xxx/Documents/xxx_repo/yy/node_modules/webpack/lib/Compilation.js:2787:11)
    at /Users/xxx/Documents/xxx_repo/yy/node_modules/webpack/lib/Compilation.js:3092:11
    at Hook.eval [as callAsync] (eval at create (/Users/xxx/Documents/xxx_repo/yy/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/Users/xxx/Documents/xxx_repo/yy/node_modules/tapable/lib/Hook.js:18:14)
    at /Users/xxx/Documents/xxx_repo/yy/node_modules/webpack/lib/Compilation.js:3085:38
    at eval (eval at create (/Users/xxx/Documents/xxx_repo/yy/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:17:1)

[daniellockyer]

Hey @stefanJi 👋🏻 I can't seem to reproduce this. What version of sqlite3 are you using? Are you able to give me a reproduction repo I can test this from? 🙂

[trydofor]

maybe sqlite3 version auto upgrade, for ~5.0.2 => 5.0.8. fix the version to 5.0.2, and following error resolved:

node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:43:28: ERROR: Could not resolve "mock-aws-s3"
node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:76:22: ERROR: Could not resolve "aws-sdk"
node_modules/@mapbox/node-pre-gyp/lib/util/s3_setup.js:112:23: ERROR: Could not resolve "nock"

[stefanJi]

I didn't do anything and it was solved

[gRoussac]

👋🏻 This can be reproduced with Webpack probably because of this package dependencies error.

mapbox/node-pre-gyp#661

which indeed seems to happen between 5.0.2 and 5.0.3

[sazzer]

I'm still getting this now, using sqlite3 5.1.4 and electron-forge 6.0.4.

And downgrading sqlite3 to 5.0.2 doesn't work because it gets HTTP 403 errors downloading the files :(

[esonwong]

I solved it by adding externals: ['nock', 'mock-aws-s3', 'aws-sdk'], in webpack config

[lwj06]

I solved it by adding externals: ['nock', 'mock-aws-s3', 'aws-sdk'], in webpack config
so do i

[paintedwood]

I made a workaround for this issue that completely removes node-pre-gyp from the packaged electron application.

I am no expert in making native packages for nodejs. I use them, as many people do, and I recently tried to package one of them, node-sqlite3, to be a part of electron app.

I use webpack and electron-forge for packaging. This tools produces a self-contained package, in the form of zip archive, debian, or rpm package. The package is OS and CPU architecture-specific (e.g. linux-x64) and it contains, as a part of electron binary, a specific version of nodejs.

In the usual setup, webpack collects the bundle which contains the node-sqlite3 package. That bundle is collected from the local files. At that point, the node-sqlite3 package already contains platform-specific binary files. As a next step, that webpack bundle is packaged along with the electron runtime to the resulting application package.

When running the resulting electron application on the end user machine, it is not expected to download any additional native binaries, compile them, cross-compile binaries to target other platforms, or upload compiled native binaries to the cloud storage.

Problem is, that functionality is a part of node-pre-gyp and it's dependencies, and all that, being the runtime dependency of node-sqlite3, is packed by the webpack to the resulting application bundle.

That packaging process is not without issues: in the current version of node-pre-gyp it may require to have a webpack configuration workaround of externals: ['aws-sdk', 'mock-aws-s3', 'nock'].

The bigger picture is, as I see it, that node-pre-gyp should not be a part of electron application bundle at all. It have it's place in the build pipeline, but carrying it further to the end-users should be avoided.

I may not see some cases when it do required, but as far as I could tell, native modules, as soon as they are installed, could work fine without node-pre-gyp.

At runtime, node-sqlite3 uses node-pre-gyp to resolve native module filename. There is a "./lib/binding/napi-v{napi_build_version}-{platform}-{libc}-{arch}" template string in package.json, it is used to locate a native module file for the specific environment.

The electron application is already packaged for a specific environment, so I believe that template string could be resolved to a static path at the time of application packaging. That will effectively remove the need to include node-pre-gyp in the electron application package.

The following code could be added to a webpack configuraton file to do exactly that.

const path = require('path');
const fs = require('fs');
const nodePreGyp = require('@mapbox/node-pre-gyp');
const webpack = require('webpack');

// The following block assumes that the current file is in the project root, so we could calculate paths as relative to __dirname
const sqliteBindingPath = nodePreGyp.find(path.join(__dirname, 'node_modules', 'sqlite3', 'package.json'));
const sqlitePatchFileName = 'sqlite3-binding-with-binary-find-patch.js';
const sqlitePatchFilePath = path.resolve(path.join(__dirname, 'node_modules', 'sqlite3', 'lib', sqlitePatchFileName));

// Here we might write an absolute path. Webpack will take care to produce a portable bundle, in which this 
// absolute path will be changes to a path, relative to the bundle itself.
fs.writeFileSync(sqlitePatchFilePath, `module.exports = require(${JSON.stringify(sqliteBindingPath)});`);

// Add this to your webpack config
module.exports = {
  plugins: [
    new webpack.NormalModuleReplacementPlugin(
      /sqlite3\/lib\/sqlite3-binding\.js$/,
      `./${sqlitePatchFileName}`
    )
  ]
};

The resulting bundle will not contain node-pre-gyp and it's dependencies. That will solve the current issue with packing (externals: ['aws-sdk' ...]) and it will make bundle lighter by 194 KB.

I hope that this workaround might be of an inspiration to someone to create a more universal solution. It could be a lightweight runtime as a part of node-pre-gyp, or, if that is not possible, as a form of some alternative lightweight project. Maybe it could be solved on a side of packages such as node-sqlite3, and not on the node-pre-gyp side. Maybe package managers such as npm could came up with some abstractions to handle native modules that could help in this regard.