Bin exported without extension, breaks node's native loader picker and ts-node/esm

[FossPrime]

Description

custom loaders like ts-node have a really hard time figuring out the mocha "binary"

The binary lacks.js or cjs extensions. Changing the node_modules/.bin/mocha extension isn't enough.

Steps to Reproduce

1. Sandbox: https://codesandbox.io/s/mocha-ts-node-16-ctt5k?file=/package.json
2. Open Terminal
3. Run npm run broken-test-npx or run source env.sh; mocha test/test.js

OR

1. npm install ts-node typescript @types/mocha mocha on a type: "modules" project.json
2. Set export NODE_OPTIONS="--loader ts-node/esm/transpile-only"
3. Run npx mocha test/any-test-at-all.js

Expected behavior:

No errors, tests run.

Actual behavior:

Setting a loader prevents node from running the mocha binary, in any way

$ npm run test # Copies mocha to mocha.js as workaround
3 passing (3ms)
$ npm run test-cjs # Copies mocha to mocha.cjs as workaround
1 passing (2ms)
$ npm run broken-test-npx # Node cant tell what loader to use and errors out
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /sandbox/node_modules/mocha/bin/mocha
$ npm run broken-test-direct # same behavior as before
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /sandbox/node_modules/mocha/bin/mocha

Reproduces how often: 100% of the time

Versions

• The output of mocha: 8.4.0
• The output of node --version: 14 (Sandbox) and 16 as well
• Your operating system
  • name and version: yes
  • architecture (32 or 64-bit): 64
• Your shell (e.g., bash, zsh, PowerShell, cmd):
• Your browser and version (if running browser tests):
• Any third-party Mocha-related modules (and their versions):
• Any code transpiler being used (and its version): ts-node loader

Additional Information

Transpile only mode does not make a difference. Extension of tests do not make a difference. As long as the loader boots up, when node proceeds to guess which loader to use, node errors out.

[juergba]

This is an working example with Mocha, ESM and ts-node/esm.
Why do you think this issue is a Mocha issue? Not a loader or configuration problem?

[FossPrime]

@juergba The error actually comes from node... internal/modules/esm/get_format.js:71:15

Again, this is just how node works - it'll use the extension loader defined or fallback on the .js loader. We can't make it use ts-node as a result.

TypeStrong/ts-node#116

Most symlinks in node_modules/.bin use extensions... I have the loader set to transpile only... it doesn't have much configuration to do.

lrwxrwxrwx   1 mind ya   21 Jun  2 13:15 mocha.js -> ../mocha/bin/mocha
lrwxrwxrwx   1 mind ya   22 Jun  2 11:45 ts-node -> ../ts-node/dist/bin.js
lrwxrwxrwx   1 mind ya   26 Jun  2 11:45 ts-node-cwd -> ../ts-node/dist/bin-cwd.js
lrwxrwxrwx   1 mind ya   29 Jun  2 11:45 ts-node-script -> ../ts-node/dist/bin-script.js
lrwxrwxrwx   1 mind ya   32 Jun  2 11:45 ts-node-transpile-only -> ../ts-node/dist/bin-transpile.js
lrwxrwxrwx   1 mind ya   40 Jun  2 11:45 ts-script -> ../ts-node/dist/bin-script-deprecated.js
lrwxrwxrwx   1 mind ya   23 May 12 09:45 eslint -> ../eslint/bin/eslint.js
lrwxrwxrwx   1 mind ya   25 May 12 09:45 esparse -> ../esprima/bin/esparse.js
lrwxrwxrwx   1 mind ya   28 May 12 09:45 esvalidate -> ../esprima/bin/esvalidate.js
lrwxrwxrwx   1 mind ya   14 May 12 09:45 flat -> ../flat/cli.js
lrwxrwxrwx   1 mind ya   19 May 12 09:45 json5 -> ../json5/lib/cli.js
lrwxrwxrwx   1 mind ya   25 May 12 09:45 js-yaml -> ../js-yaml/bin/js-yaml.js
lrwxrwxrwx   1 mind ya   15 May 12 09:45 katex -> ../katex/cli.js
lrwxrwxrwx   1 mind ya   33 May 12 09:45 markdown-it -> ../markdown-it/bin/markdown-it.js
lrwxrwxrwx   1 mind ya   14 May 12 09:45 mime -> ../mime/cli.js
lrwxrwxrwx   1 mind ya   20 May 12 09:45 mkdirp -> ../mkdirp/bin/cmd.js
lrwxrwxrwx   1 mind ya   24 May 12 09:45 nanoid -> ../nanoid/bin/nanoid.cjs
lrwxrwxrwx   1 mind ya   29 May 12 09:45 replace-in-file -> ../replace-in-file/bin/cli.js
lrwxrwxrwx   1 mind ya   16 May 12 09:45 rimraf -> ../rimraf/bin.js
lrwxrwxrwx   1 mind ya   23 May 12 09:45 semver -> ../semver/bin/semver.js
lrwxrwxrwx   1 mind ya   19 May 12 09:45 shjs -> ../shelljs/bin/shjs

nandoid goes as far as using cjs

[juergba]

Again, this is just how node works - it'll use the extension loader defined or fallback on the .js loader. We can't make it use ts-node as a result.

I haven't understood yet, or something is wrong in your argumentation.

• without extension the above fallback should be triggered. Which would be correct with .js loader. Why isn't this the case?
• why do you pass Mocha, a CJS module, to your ESM loader? For transpilation?
• why does my sample mentioned above work? What is the difference to your case?

[juergba]

If you remove --loader ... from NODE_OPTIONS and add it to .mocharc.cjs instead , does it work then?
I guess so, because in this case Mocha starts first, then the loader is called in a subprocess.. The entry point is lib/cli/cli.js.

[juergba]

Yes, above sample does not work anymore, when I remove loader: 'ts-node/esm' from .mocharc.cjs and use NODE_OPTIONS="--loader=ts-node/esm" instead.

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for D:\temp\ts-node-repros\node_modules\mocha\bin\mocha
    at defaultGetFormat (internal/modules/esm/get_format.js:71:15)
    at defer (D:\temp\ts-node-repros\node_modules\ts-node\src\esm.ts:81:7)
    at D:\temp\ts-node-repros\node_modules\ts-node\src\esm.ts:103:12
    at Generator.next (<anonymous>)
    at D:\temp\ts-node-repros\node_modules\ts-node\dist\esm.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (D:\temp\ts-node-repros\node_modules\ts-node\dist\esm.js:4:12)
    at getFormat (D:\temp\ts-node-repros\node_modules\ts-node\dist\esm.js:54:16)
    at Loader.getFormat (internal/modules/esm/loader.js:102:42)
    at Loader.getModuleJob (internal/modules/esm/loader.js:231:31)

@cspotcode can you have a look at this, please?

[cspotcode]

This is a confirmed bug in nodejs. Please remind them that they need to fix it! I have cited mocha as one of the libraries being broken by it, but it will help if mocha's users and maintainers also remind them (politely) that they need to fix this!
nodejs/node#33226

To reproduce the bug, use a completely empty --loader. This will prove that node is responsible for the error, not ts-node nor any other loader.

$ node -v
v16.2.0

$ touch empty-loader.mjs
$ touch extensionless-entrypoint
$ node --loader empty-loader.mjs ./extensionless-entrypoint
(node:10077) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
node:internal/process/esm_loader:74
    internalBinding('errors').triggerUncaughtException(
                              ^

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /c/Users/cspot/extensionless-entrypoint
    at new NodeError (node:internal/errors:363:5)
    at Loader.defaultGetFormat [as _getFormat] (node:internal/modules/esm/get_format:71:15)
    at Loader.getFormat (node:internal/modules/esm/loader:105:42)
    at Loader.getModuleJob (node:internal/modules/esm/loader:243:31)
    at async Loader.import (node:internal/modules/esm/loader:177:17)
    at async Object.loadESM (node:internal/process/esm_loader:68:5) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}

EDIT Fixed a typo above; the node --loader invocation was missing the ./extensionless-entrypoint

[juergba]

@cspotcode thank you for your explanation, I will remind nodejs tomorrow.

@rayfoss you have the workaround with .mocharc.cjs. I don't know yet what to do with our extensionless binary.

[cspotcode]

I think if mocha wants to implement a simple fix, they can add a new bin entrypoint file with a file extension (make sure to keep the old one, too, for backwards-compatibility) and then point package.json "bin" field to the new entrypoint with the file extension.

I believe that update can be published as a non-breaking change.

[FossPrime]

I think if mocha wants to implement a simple fix, they can add a new bin entrypoint file with a file extension (make sure to keep the old one, too, for backwards-compatibility) and then point package.json "bin" field to the new entrypoint with the file extension.

I believe that update can be published as a non-breaking change.

... My PR is not a breaking change in any way. mocha/bin/_mocha.js is a new entry point. And NPM makes a symlink called .bin/_mocha. People who use mocha/bin/_mocha directly or via the symlink .bin/_mocha would be unaffected.

There is an argument to be made that it should be mocha/bin/mocha.js, not mocha/bin/_mocha.js, which is then symlinked to .bin/_mocha by NPM, I was just preserving the convention established by mocha/bin/_mocha.

[cspotcode]

You mention a drawback in your PR; is that non-breaking?

We might be talking about the same thing. I'm imagining that mocha ships 4x different files: mocha, mocha.js, _mocha, and _mocha.js. package.json "bin" points to the ones with file extensions.

[FossPrime]

You mention a drawback in your PR; is that non-breaking?

We might be talking about the same thing. I'm imagining that mocha ships 4x different files: mocha, mocha.js, _mocha, and _mocha.js. package.json "bin" points to the ones with file extensions.

My PR only addresses the issue at hand, I couldn't see how you guys output bundled code... So the PR has

• _mocha (naked vanilla)
• _mocha.js (clothed vanilla)
• mocha (naked bundled)

Symbolic Links made by NPM:

• .bin/_mocha (& to clothed vanilla _mocha.js)
• .bin/mocha (& to naked bundled mocha)

We can make a mocha/bin/mocha.js that uses the naked bundled mocha/bin/mocha without duplicating the big file by using a funky, but legal, side-effect only, require or import call to the naked bundle.

Update: I corrected the description of the PR to reflect it's contents. I updated the PR code a few hours after publishing to duplicate naked vanilla mocha/bin/_mocha once I stumped on documentation and projects that directly referenced mocha/.bin/_mocha... And remembered Deno and custom package managers exist, and they may not read our package.json for the location of our stable endpoints.

[FossPrime]

no longer an issue in Mocha 9

[lachrist]

Hi again @juergba, could it be that the loader mocha option override require? Sorry, I could not find any documentation on the loader option...

[juergba]

@lachrist There is no loader option in Mocha.

Just use 'experimental-loader' : './lib/loader.mjs'. Or with the alias name: loader: './lib/loader.mjs'

Edit: Mocha recognizes Node's option and forwards them to Node by spawning a child-process.

[lachrist]

Now, it makes more sense! I got confused because node has no documented loader option, hence I assumed it was a mocha option :|

Thanks for the swift answer!

[juergba]

Welcome.
Run node -h, then: --loader, --experimental-loader=... use the specified module as a custom loader

[JakobJingleheimer]

@juergba @giltayar This issue occurs today. I'm not sure why it was fixed in mocha v9 release but occurs now exactly as OP described for exactly the reason OP described.

I spoke with npm and yarn authors, and both package managers automatically alias a non-naked bin with a naked version (but not the other way around).

There is an on-going discussion within Node.js on how to handle naked bin files, but it has become rather protracted and does not appear to be resolving any time soon.

The quickest solution to get mocha working with loaders is to:

• add a .js file extension to the currently naked bin file
• add "type": "commonjs" to mocha's package.json (commonjs is the default, but it saves Node.js having to look further for the info and only eventually fall back to the default value, and is future-proofing for when Node.js switches the default to module)

I'd be happy to do this if you agree.

[juergba]

@JakobJingleheimer no, I'm against this .js solution. We haven't fixed anything in v9 either.

When you set --loader with NODE_OPTIONS then it fails, because Mocha's entry point is extensionless.
When you set --loader via Mocha options (mocha --loader=... or mocha -n loader=...) then Mocha launches a child-process and the entry point lib/cli/cli.js has an extension.

[JakobJingleheimer]

Ah. Perhaps a documentation update then?

[giltayar]

@juergba why are you against this? (or at least two bin entry points: one without an extension and one with it)

[JakobJingleheimer]

Also keeping in mind that npm and yarn both handle supplying the naked one automatically.

[juergba]

@giltayar --loader is still experimental and I don't see any reason to adapt Mocha at that floating stage. Furthermore we have this easy work-around.

In aprox. April/May we will publish Mocha v10. If Node still hasn't made up their mind until then about this extensionless binary story, we evtl. could add *.js to our binary as a breaking change.

[lachrist]

Hi guys I've discussed this issue with the node team. And this will probably never be solved on their end. Checkout nodejs/node#41465. In particular, this is my final take on it:

At the moment, monitoring ECMAScript modules requires --experimental-loader flag. But this flag makes node always use the new esm loader to load the main file. Without this flag, node will sometimes use the old cjs loader. To be clear, the new esm loader is capable of loading commonjs files as well (along with native modules, web assembly modules, ...). So most of the time this change should not be noticeable. The biggest difference I encountered so far is that the cjs loader handles files without extension whereas the esm loader would throw an exception. This is intentional: the cjs loader only has to handle one kind of module (commonjs modules) whereas the esm loader has to handle different kinds of modules and the node team wants to reserve extension-less files for web assembly.

[juergba]

What is the (historic) reason that binaries (or executables?) are extensionless?
The first line #!/usr/bin/env node means, this file is an executable, right? In Linux only? In Windows this line has no purpose.

Also keeping in mind that npm and yarn both handle supplying the naked one automatically.

For which reason does npm/yarn do that?

[JakobJingleheimer]

Yes, I'm 99% sure the historic reason is days of yore (unix/linux).

Note that adding the .js file extension would NOT be a breaking change as npm and yarn will automatically handle the extension-less part for you.

Also, I'm completely confident that making the two tiny changes I suggested will 100% be compatible now and in future, wherever loaders lands ;)

@lachrist cough I am on the node team 🙂

There is a more recent discussion, but I'm not linking to it at the moment because I need to port it from a PR (that I need to closed) to an actual discussion.

[cspotcode]

The first line #!/usr/bin/env node means, this file is an executable, right? In Linux only?

Technically the execute bit tells Linux it's an executable, and the two-byte sequence #! is a magic number understood by the Linux kernel, used to differentiate different binary formats. That's my understanding at least.

Package managers use https://www.npmjs.com/package/@zkochan/cmd-shim to generate .ps1 and .cmd shims that achieve the same effect on Windows.

[cspotcode]

#4645 (comment)

^^ This is a fully backwards-compatible fix and takes little effort, right? I feel like there's some confusion with talk of this being a breaking change, but it's not. Is there anything we can explain further?

[juergba]

I spoke with npm and yarn authors, and both package managers automatically alias a non-naked bin with a naked version (but not the other way around).

If you mean: bin/mocha.js results in : .bin/mocha then I agree. Otherwise ..?

I can live with adding an additional bin/mocha.js requiring the naked one. The second binary bin/_mocha is on sneaky deprecation, we should leave it as is.

@JakobJingleheimer please go ahead with your PR, if you are still on fire. Otherwise we wait till Mocha@10.

[JakobJingleheimer]

I spoke with npm and yarn authors, and both package managers automatically alias a non-naked bin with a naked version (but not the other way around).

If you mean: bin/mocha.js results in : .bin/mocha then I agree. Otherwise ..?

Yes, exactly 🙂

I can live with adding an additional bin/mocha.js requiring the naked one. The second binary bin/_mocha is on sneaky deprecation, we should leave it as is.

You don't have to do this: npm and yarn will take care of it automatically.

@JakobJingleheimer please go ahead with your PR, if you are still on fire. Otherwise we wait till Mocha@10.

Not on fire. I've just seen quite a few bugs about it. Whatever you prefer 🙂

I've consolidated the discussions into nodejs/node#41711

[juergba]

You don't have to do this: npm and yarn will take care of it automatically.

This is not true. Take a look at e.g. nanoid after installation with npm, there is a bin/nanoid.cjs but no bin/nanoid.

[cspotcode]

Yeah some tools have hardcoded paths to the extensionless JS file, so just as easy to keep it there and add an extension-ed file that requires it. (Or vice versa, either works)

…

On Thu, Jan 27, 2022, 1:27 PM Juerg B. ***@***.***> wrote: You don't have to do this: npm and yarn will take care of it automatically. This is not true. Take a look at e.g. nanoid after installation with npm, there is a bin/nanoid.cjs but no bin/nanoid. — Reply to this email directly, view it on GitHub <#4645 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC35OCZCFLDEB42M26PVULUYGFANANCNFSM46AJASBQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[cspotcode]

ts-node 10.6.0 implements a workaround that hopefully mimics the bugfix which will eventually be published in node core. This means that, for users of mocha and ts-node, upgrading to the latest ts-node should avoid this issue.

https://github.com/TypeStrong/ts-node/releases/tag/v10.6.0