ProjectReferences: ts-loader doesnt build dependent projects

[jleider]

Expected Behaviour

Based on this highly anticipated and recently merged PR: #935 by @sheetalkamat I would expect ts-loader to build dependent projects if projectReferences: true.

Actual Behaviour

Dependent project reference is not built when building project.

The test projectReferencesNotBuilt referenced in the "steps to reproduce" section seems to account for the fact that the dependent project may not already be built. However, the expected output of that test is just the TypeScript error saying the project hasn't been built and to run tsc --build to build the dependent project. I would expect the projectReferences config option to basically run the API equivalent of tsc --build.

Actual application error:

Module build failed (from ../node_modules/ts-loader/index.js):
Error: Could not find output JavaScript file for input assets/scripts/api/api.ts (looked at assets/scripts/lib/api/api.js).
The input file is part of a project reference located at tsconfig.json, so ts-loader is looking for the project’s pre-built output on disk. Try running `tsc --build` to build project references.
    at successLoader (/src/node_modules/ts-loader/dist/index.js:46:19)
    at Object.loader (/src/node_modules/ts-loader/dist/index.js:22:12)
 @ ./portal/assets/scripts/actions/Dashboard.tsx 28:0-129 42:15-37 43:19-37 44:23-45 45:16-41
 @ ./portal/assets/scripts/router.ts
 @ ./portal/assets/scripts/index.tsx
./assets/scripts/api/methods.ts

Steps to Reproduce the Problem

Typescript: v3.6.3
ts-loader: v6.1.0

yarn run comparison-tests --single-test projectReferencesNotBuilt                                                    09/17 15:40:15 121.372s
yarn run v1.17.3
$ tsc --project "./test/comparison-tests" && npm link ./test/comparison-tests/testLib && node test/comparison-tests/run-tests.js --single-test projectReferencesNotBuilt
npm WARN lib@0.0.1 No repository field.
npm WARN lib@0.0.1 No license field.

up to date in 0.276s
found 0 vulnerabilities

/Users/justin/.fnm/node-versions/v12.3.1/installation/lib/node_modules/lib -> /src/ts-loader/test/comparison-tests/testLib
/src/ts-loader/node_modules/lib -> /Users/justin/.fnm/node-versions/v12.3.1/installation/lib/node_modules/lib -> /src/ts-loader/test/comparison-tests/testLib

-------------------------------------------------------------------------

Starting to run test suites...



  projectReferencesNotBuilt
    1) should have the correct output
    ✓ should work with transpileOnly (292ms)


  1 passing (2s)
  1 failing

  1) projectReferencesNotBuilt
       should have the correct output:

      Uncaught AssertionError [ERR_ASSERTION]: bundle.js is different between actual and expected
      + expected - actual

       /*!**********!*/
         !*** ./lib/index.ts ***!
         /**********/
       /*! no static exports found */
      -/***/ (function(module, exports, __webpack_require__) {
      -"use strict";
      -eval("
      -exports.__esModule = true;
      -exports.lib = {
      -    one: 1,
      -    two: 2,
      -    three: 3
      -};
      +/***/ (function(module, exports) {
      +eval("throw new Error(/"Module build failed (fromts-loader):/nError: Could not find output JavaScript file for input lib/index.ts (looked at lib/index.js)./nThe input file is part of a project reference located at lib/tsconfig.json, so ts-loader is looking for the project’s pre-built output on disk. Try running `tsc --build` to build project references./n    at successLoader (ts-loader)/n    at Object.loader (ts-loader)/");
       /# sourceURL=webpack://./lib/index.ts?");
       /***/ })
       /******/ });

      at compareActualAndExpected (test/comparison-tests/create-and-execute-test.js:467:16)
      at /src/ts-loader/test/comparison-tests/create-and-execute-test.js:333:13
      at Array.forEach (<anonymous>)
      at compareFiles (test/comparison-tests/create-and-execute-test.js:329:31)
      at Watching.handler (test/comparison-tests/create-and-execute-test.js:194:9)
      at /src/ts-loader/node_modules/webpack/lib/Watching.js:100:9
      at AsyncSeriesHook.eval [as callAsync] (eval at create (node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:32:10), <anonymous>:6:1)
      at AsyncSeriesHook.lazyCompileHook (node_modules/webpack/node_modules/tapable/lib/Hook.js:154:20)
      at Watching._done (node_modules/webpack/lib/Watching.js:99:28)
      at /src/ts-loader/node_modules/webpack/lib/Watching.js:73:19
      at Compiler.emitRecords (node_modules/webpack/lib/Compiler.js:366:39)
      at /src/ts-loader/node_modules/webpack/lib/Watching.js:54:20
      at /src/ts-loader/node_modules/webpack/lib/Compiler.js:352:14
      at AsyncSeriesHook.eval [as callAsync] (eval at create (node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:32:10), <anonymous>:6:1)
      at AsyncSeriesHook.lazyCompileHook (node_modules/webpack/node_modules/tapable/lib/Hook.js:154:20)
      at /src/ts-loader/node_modules/webpack/lib/Compiler.js:349:27
      at done (node_modules/neo-async/async.js:2854:11)
      at /src/ts-loader/node_modules/neo-async/async.js:2805:7
      at /src/ts-loader/node_modules/graceful-fs/graceful-fs.js:45:10
      at FSReqCallback.oncomplete (fs.js:137:23)

[johnnyreilly]

There's work in progress by @sheetalkamat here:

#1003

It might be worth testing her PR and seeing if that fixes you. We'd love to hear back if it does

[johnnyreilly]

Should be fixed with V6.1.1

[jleider]

I see that the tests have been updated: https://github.com/TypeStrong/ts-loader/pull/1003/files#diff-18ab58ce03516261da5fb32ec3f05984R108 and I dont get that error anymore but I also still get all the other definition files have not been built from source errors from referenced projects. Is there a new config option I'm missing?

Output file 'assets/scripts/lib/util/exhaustive.d.ts' has not been built from source file '/assets/scripts/util/exhaustive.ts'.

@johnnyreilly @sheetalkamat this still seems to be an issue with v6.1.1. Could you please reopen?

[jleider]

I pulled down the latest updates locally and ran the tests again. I am still getting the following test failures.

Additionally, one thing I have noticed about the tests is that they are missing the declaration: true option that is specified as required for project references: https://www.typescriptlang.org/docs/handbook/project-references.html#composite

yarn run comparison-tests --single-test projectReferencesNotBuilt                                                                09/20 12:50:32 7.927s
yarn run v1.17.3
$ git clean -xfd test/comparison-tests && tsc --project "./test/comparison-tests" && npm link ./test/comparison-tests/testLib && node test/comparison-tests/run-tests.js --single-test projectReferencesNotBuilt
npm WARN lib@0.0.1 No repository field.
npm WARN lib@0.0.1 No license field.

up to date in 0.574s
found 0 vulnerabilities

/Users/justin/.fnm/node-versions/v11.15.0/installation/lib/node_modules/lib -> /src/ts-loader/test/comparison-tests/testLib
/src/ts-loader/node_modules/lib -> /Users/justin/.fnm/node-versions/v11.15.0/installation/lib/node_modules/lib -> /src/ts-loader/test/comparison-tests/testLib

-------------------------------------------------------------------------

Starting to run test suites...



  projectReferencesNotBuilt
    1) should have the correct output
    2) should work with transpileOnly


  0 passing (2s)
  2 failing

  1) projectReferencesNotBuilt
       should have the correct output:

      Uncaught AssertionError [ERR_ASSERTION]: output.txt is different between actual and expected
      + expected - actual

           Asset     Size  Chunks             Chunk Names
      -bundle.js A-NUMBER-OF KiB main  [emitted]  main
      +                  bundle.js A-NUMBER-OF KiB main  [emitted]  main
      +          ../lib/index.d.ts A-NUMBER-OF bytes [emitted]
      +../lib/tsconfig.tsbuildinfo A-NUMBER-OF KiB [emitted]
       Entrypoint main = bundle.js
       [./app.ts] A-NUMBER-OF bytes {main} [built]
      -[./lib/index.ts] A-NUMBER-OF bytes {main} [built]
      -ERROR in tsconfig.json
      -[tsl] ERROR
      -      TS6305: Output file 'index.d.ts' has not been built from source file 'index.ts'.
      +[./lib/index.ts] A-NUMBER-OF bytes {main} [built]

      at compareActualAndExpected (test/comparison-tests/create-and-execute-test.js:470:16)
      at /src/ts-loader/test/comparison-tests/create-and-execute-test.js:338:13
      at Array.forEach (<anonymous>)
      at compareFiles (test/comparison-tests/create-and-execute-test.js:334:31)
      at Watching.handler (test/comparison-tests/create-and-execute-test.js:199:9)
      at compiler.hooks.done.callAsync (node_modules/webpack/lib/Watching.js:100:9)
      at AsyncSeriesHook.eval [as callAsync] (eval at create (node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:32:10), <anonymous>:6:1)
      at AsyncSeriesHook.lazyCompileHook (node_modules/webpack/node_modules/tapable/lib/Hook.js:154:20)
      at Watching._done (node_modules/webpack/lib/Watching.js:99:28)
      at compiler.emitRecords.err (node_modules/webpack/lib/Watching.js:73:19)
      at Compiler.emitRecords (node_modules/webpack/lib/Compiler.js:366:39)
      at compiler.emitAssets.err (node_modules/webpack/lib/Watching.js:54:20)
      at hooks.afterEmit.callAsync.err (node_modules/webpack/lib/Compiler.js:352:14)
      at AsyncSeriesHook.eval [as callAsync] (eval at create (node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:32:10), <anonymous>:6:1)
      at AsyncSeriesHook.lazyCompileHook (node_modules/webpack/node_modules/tapable/lib/Hook.js:154:20)
      at asyncLib.forEach.err (node_modules/webpack/lib/Compiler.js:349:27)
      at done (node_modules/neo-async/async.js:2854:11)
      at /src/ts-loader/node_modules/neo-async/async.js:2805:7
      at /src/ts-loader/node_modules/graceful-fs/graceful-fs.js:45:10
      at FSReqCallback.args [as oncomplete] (fs.js:145:20)

  2) projectReferencesNotBuilt
       should work with transpileOnly:

      Uncaught AssertionError [ERR_ASSERTION]: bundle.js is different between actual and expected
      + expected - actual

       /*!**********!*/
         !*** ./lib/index.ts ***!
         /**********/
       /*! no static exports found */
      -/***/ (function(module, exports) {
      -eval("throw new Error(/"Module build failed (fromts-loader):/nError: Could not find output JavaScript file for input lib/index.ts (looked at lib/index.js)./nThe input file is part of a project reference located at lib/tsconfig.json, so ts-loader is looking for the project’s pre-built output on disk. Try running `tsc --build` to build project references./n    at successLoader (ts-loader)/n    at Object.loader (ts-loader)/");
      +/***/ (function(module, exports, __webpack_require__) {
      +"use strict";
      +eval("
      +Object.defineProperty(exports, /"__esModule/", { value: true });
      +exports.lib = {
      +    one: 1,
      +    two: 2,
      +    three: 3
      +};
       /# sourceURL=webpack://./lib/index.ts?");
       /***/ })
       /******/ });

      at compareActualAndExpected (test/comparison-tests/create-and-execute-test.js:470:16)
      at /src/ts-loader/test/comparison-tests/create-and-execute-test.js:338:13
      at Array.forEach (<anonymous>)
      at compareFiles (test/comparison-tests/create-and-execute-test.js:334:31)
      at Watching.handler (test/comparison-tests/create-and-execute-test.js:199:9)
      at compiler.hooks.done.callAsync (node_modules/webpack/lib/Watching.js:100:9)
      at AsyncSeriesHook.eval [as callAsync] (eval at create (node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:32:10), <anonymous>:6:1)
      at AsyncSeriesHook.lazyCompileHook (node_modules/webpack/node_modules/tapable/lib/Hook.js:154:20)
      at Watching._done (node_modules/webpack/lib/Watching.js:99:28)
      at compiler.emitRecords.err (node_modules/webpack/lib/Watching.js:73:19)
      at Compiler.emitRecords (node_modules/webpack/lib/Compiler.js:366:39)
      at compiler.emitAssets.err (node_modules/webpack/lib/Watching.js:54:20)
      at hooks.afterEmit.callAsync.err (node_modules/webpack/lib/Compiler.js:352:14)
      at AsyncSeriesHook.eval [as callAsync] (eval at create (node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:32:10), <anonymous>:6:1)
      at AsyncSeriesHook.lazyCompileHook (node_modules/webpack/node_modules/tapable/lib/Hook.js:154:20)
      at asyncLib.forEach.err (node_modules/webpack/lib/Compiler.js:349:27)
      at done (node_modules/neo-async/async.js:2854:11)
      at /src/ts-loader/node_modules/neo-async/async.js:2805:7
      at /src/ts-loader/node_modules/graceful-fs/graceful-fs.js:45:10
      at FSReqCallback.args [as oncomplete] (fs.js:145:20)

[johnnyreilly]

Thanks for the details - @sheetalkamat it seems there might still be some issues.

@jleider what os are you running? Our CI runs on Linux and Windows so I'm kind of surprised you still have failing tests..

[sheetalkamat]

@jleider the lib folder in that test has composite set which is referenced in the project. What is your typescript version that you are using. Are you at master and latest commit for ts-loader

[jleider]

I am using MacOS v10.14.6. I tested with both Node v11.15 and v12.8. TypeScript version v3.6.3 for my own project and whatever ts-loader requires for the tests. I sync'ed my own fork of ts-loader from the upstream TypeStrong/ts-loader and pulled the latest master then ran yarn install --force right before running the tests locally.

[jleider]

Also, I suspected that fork-ts-checker-webpack-plugin could be causing issues in my own project so I removed that from my webpack config and tried with transpileOnly: false and get this TypeScript error which seems like an actual TSC compiler bug when I google for the error.

[12:40:11] Error: Debug Failure. False expression.
    at Object.getOutputDeclarationFileName (/src/node_modules/typescript/lib/typescript.js:86168:18)
    at getOutputFileNames (/src/node_modules/ts-loader/dist/instances.js:262:43)
    at /src/node_modules/ts-loader/dist/instances.js:281:13
    at worker (/src/node_modules/ts-loader/dist/instances.js:211:32)
    at forEachResolvedProjectReference (/src/node_modules/ts-loader/dist/instances.js:198:12)
    at getOutputFilesFromReference (/src/node_modules/ts-loader/dist/instances.js:273:12)
    at Object.getEmitOutput (/src/node_modules/ts-loader/dist/instances.js:302:37)
    at provideDeclarationFilesToWebpack (/src/node_modules/ts-loader/dist/after-compile.js:185:41)
    at /src/node_modules/ts-loader/dist/after-compile.js:24:9
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/src/web/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)

[sheetalkamat]

The assert is fixed by #1015

[jleider]

@sheetalkamat I have three things to report:

1. Between the 6.1.1 and 6.2.2 changes this is mostly working! However, there is still one issue that seems to be a race condition. The first time a project is built which contains a project reference, the referenced project will be built on the first run but the exported .d.ts files are not "present" for the compile of the parent project until the second run.

2. If there are any .tsbuildinfo files hanging around from a previous build but there are no exported files due to being removed from the file system the .d.ts files will never be rebuilt. I am not sure if this is a ts-loader issue or if that is expected behavior of the tsc compiler in which case a user must tsc --force, tsc --clean the project or remove the .tsbuildinfo files first.

3. There is an unexpected behavior of the placement of the .tsbuildinfo file. By default according to the ts compiler options, the .tsbuildinfo file should be located in the same directory as the tsconfig file. However, I see it being written to disk one level up from my rootDir or two directories up from my outDir. For example, if rootDir: "./assets/scripts and outDir: "./assets/scripts/lib" then the .tsbuildinfo file is written to ./assets/tsconfig.tsbuildinfo instead of what I would expect to be ./tsconfig.tsbuildinfo. Setting the tsBuildInfoFile: "./tsconfig.tsbuildinfo" directly gets around this issue but I don't think should be necessary.

[sheetalkamat]

1> Repro is needed since I don't see this when I run test locally
2> As you guessed that is working as intended and same as what tsc does.
3> Again repro is needed to investigate.

[jleider]

The repro for 1> is the existing test which fails: yarn run comparison-tests --single-test projectReferencesNotBuilt

yarn run comparison-tests --single-test projectReferencesNotBuilt                                                                                         09/23 13:36:23 115.637s
yarn run v1.17.3
$ git clean -xfd test/comparison-tests && tsc --project "./test/comparison-tests" && npm link ./test/comparison-tests/testLib && node test/comparison-tests/run-tests.js --single-test projectReferencesNotBuilt
npm WARN lib@0.0.1 No repository field.
npm WARN lib@0.0.1 No license field.

up to date in 0.353s
found 0 vulnerabilities

/Users/justin/.fnm/node-versions/v12.3.1/installation/lib/node_modules/lib -> /src/ts-loader/test/comparison-tests/testLib
/src/ts-loader/node_modules/lib -> /Users/justin/.fnm/node-versions/v12.3.1/installation/lib/node_modules/lib -> /src/ts-loader/test/comparison-tests/testLib

-------------------------------------------------------------------------

Starting to run test suites...



  projectReferencesNotBuilt
    1) should have the correct output
    2) should work with transpileOnly


  0 passing (2s)
  2 failing

  1) projectReferencesNotBuilt
       should have the correct output:

      Uncaught AssertionError [ERR_ASSERTION]: output.txt is different between actual and expected
      + expected - actual

           Asset     Size  Chunks             Chunk Names
      -bundle.js A-NUMBER-OF KiB main  [emitted]  main
      +                  bundle.js A-NUMBER-OF KiB main  [emitted]  main
      +          ../lib/index.d.ts A-NUMBER-OF bytes [emitted]
      +../lib/tsconfig.tsbuildinfo A-NUMBER-OF KiB [emitted]
       Entrypoint main = bundle.js
       [./app.ts] A-NUMBER-OF bytes {main} [built]
      -[./lib/index.ts] A-NUMBER-OF bytes {main} [built]
      -ERROR in tsconfig.json
      -[tsl] ERROR
      -      TS6305: Output file 'index.d.ts' has not been built from source file 'index.ts'.
      +[./lib/index.ts] A-NUMBER-OF bytes {main} [built]

      at compareActualAndExpected (test/comparison-tests/create-and-execute-test.js:470:16)
      at /src/ts-loader/test/comparison-tests/create-and-execute-test.js:338:13
      at Array.forEach (<anonymous>)
      at compareFiles (test/comparison-tests/create-and-execute-test.js:334:31)
      at Watching.handler (test/comparison-tests/create-and-execute-test.js:199:9)
      at /src/ts-loader/node_modules/webpack/lib/Watching.js:100:9
      at AsyncSeriesHook.eval [as callAsync] (eval at create (node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:32:10), <anonymous>:6:1)
      at AsyncSeriesHook.lazyCompileHook (node_modules/webpack/node_modules/tapable/lib/Hook.js:154:20)
      at Watching._done (node_modules/webpack/lib/Watching.js:99:28)
      at /src/ts-loader/node_modules/webpack/lib/Watching.js:73:19
      at Compiler.emitRecords (node_modules/webpack/lib/Compiler.js:366:39)
      at /src/ts-loader/node_modules/webpack/lib/Watching.js:54:20
      at /src/ts-loader/node_modules/webpack/lib/Compiler.js:352:14
      at AsyncSeriesHook.eval [as callAsync] (eval at create (node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:32:10), <anonymous>:6:1)
      at AsyncSeriesHook.lazyCompileHook (node_modules/webpack/node_modules/tapable/lib/Hook.js:154:20)
      at /src/ts-loader/node_modules/webpack/lib/Compiler.js:349:27
      at done (node_modules/neo-async/async.js:2854:11)
      at /src/ts-loader/node_modules/neo-async/async.js:2805:7
      at /src/ts-loader/node_modules/graceful-fs/graceful-fs.js:45:10
      at FSReqCallback.oncomplete (fs.js:137:23)

  2) projectReferencesNotBuilt
       should work with transpileOnly:

      Uncaught AssertionError [ERR_ASSERTION]: bundle.js is different between actual and expected
      + expected - actual

       /*!**********!*/
         !*** ./lib/index.ts ***!
         /**********/
       /*! no static exports found */
      -/***/ (function(module, exports) {
      -eval("throw new Error(/"Module build failed (fromts-loader):/nError: Could not find output JavaScript file for input lib/index.ts (looked at lib/index.js)./nThe input file is part of a project reference located at lib/tsconfig.json, so ts-loader is looking for the project’s pre-built output on disk. Try running `tsc --build` to build project references./n    at successLoader (ts-loader)/n    at Object.loader (ts-loader)/");
      +/***/ (function(module, exports, __webpack_require__) {
      +"use strict";
      +eval("
      +Object.defineProperty(exports, /"__esModule/", { value: true });
      +exports.lib = {
      +    one: 1,
      +    two: 2,
      +    three: 3
      +};
       /# sourceURL=webpack://./lib/index.ts?");
       /***/ })
       /******/ });

      at compareActualAndExpected (test/comparison-tests/create-and-execute-test.js:470:16)
      at /src/ts-loader/test/comparison-tests/create-and-execute-test.js:338:13
      at Array.forEach (<anonymous>)
      at compareFiles (test/comparison-tests/create-and-execute-test.js:334:31)
      at Watching.handler (test/comparison-tests/create-and-execute-test.js:199:9)
      at /src/ts-loader/node_modules/webpack/lib/Watching.js:100:9
      at AsyncSeriesHook.eval [as callAsync] (eval at create (node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:32:10), <anonymous>:6:1)
      at AsyncSeriesHook.lazyCompileHook (node_modules/webpack/node_modules/tapable/lib/Hook.js:154:20)
      at Watching._done (node_modules/webpack/lib/Watching.js:99:28)
      at /src/ts-loader/node_modules/webpack/lib/Watching.js:73:19
      at Compiler.emitRecords (node_modules/webpack/lib/Compiler.js:366:39)
      at /src/ts-loader/node_modules/webpack/lib/Watching.js:54:20
      at /src/ts-loader/node_modules/webpack/lib/Compiler.js:352:14
      at AsyncSeriesHook.eval [as callAsync] (eval at create (node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:32:10), <anonymous>:6:1)
      at AsyncSeriesHook.lazyCompileHook (node_modules/webpack/node_modules/tapable/lib/Hook.js:154:20)
      at /src/ts-loader/node_modules/webpack/lib/Compiler.js:349:27
      at done (node_modules/neo-async/async.js:2854:11)
      at /src/ts-loader/node_modules/neo-async/async.js:2805:7
      at /src/ts-loader/node_modules/graceful-fs/graceful-fs.js:45:10
      at FSReqCallback.oncomplete (fs.js:137:23)

[sheetalkamat]

But the test doesn't fail either on CI or on my machine so that's not a repro at my end....