Node.js syntax error when trying to run myst-to-md example

[astrojuanlu]

This is my first time using Node.js so please forgive this totally noob question 😇

I'm trying to run the myst-to-md example that appears in the README https://github.com/executablebooks/mystjs/blob/main/packages/myst-to-md/docs/index.md#simple-example

> cat index.js
...  // Copy-paste from above
> cat package.json                                       (unified) 
{
  "name": "unified-experiments",
  "version": "1.0.0",
  "description": "",
  "type": "module",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "myst-parser": "^0.0.22",
    "myst-to-md": "^0.0.2",
    "rehype-stringify": "^9.0.3",
    "remark-parse": "^10.0.1",
    "remark-rehype": "^10.1.0",
    "remark-slug": "^7.0.1",
    "remark-stringify": "^10.0.2",
    "remark-toc": "^8.0.1",
    "unified": "^10.1.2",
    "unified-stream": "^2.0.0"
  }
}

But I'm getting a SyntaxError:

> node --trace-warnings index.js                             (unified) 
(node:17287) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
/Users/juan_cano/Projects/Personal/unified-experiments/node_modules/myst-to-md/dist/esm/index.js:1
import { gfmFootnoteToMarkdown } from 'mdast-util-gfm-footnote';
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at internalCompileFunction (node:internal/vm:73:18)
    at wrapSafe (node:internal/modules/cjs/loader:1176:20)
    at Module._compile (node:internal/modules/cjs/loader:1218:27)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Module._load (node:internal/modules/cjs/loader:958:12)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:169:29)
    at ModuleJob.run (node:internal/modules/esm/module_job:194:25)

Node.js v18.15.0

There are of course numerous references in StackOverflow and the like to this problem, but my package.json already contains "type": "module" and looks like the error is coming from a dependency, so I'm not sure how to proceed.

[rowanc1]

Hi @astrojuanlu,

I put together a repo and video tutorial here:
https://github.com/rowanc1/myst-playground

The node ecosystem can be really tough to get up and running with, so hopefully that template can cut through some of the noise and get you to actually playing with something a bit faster!

This uses esbuild to bundle everything up into a single file that you can then execute directly in node. There are a lot of other ways to do this; this one is both fast, and ... generally just works!?

I put a few demos in that repo, and created a rough walkthrough video, that maybe can get you up and running. Maybe that repo turns into something more with some prompting from you!?

Best of luck, happy to answer more questions!!

[astrojuanlu]

A million thanks @rowanc1, this is very much appreciated! 🎉 Especially the 5 minute video helped me understand a bit some of the key concepts.

...However, I was still not entirely satisfied with having to rely on some boilerplate I didn't fully understand, and your repository has allowed me to more fully comprehend the situation. Allow me to write another comment with what I think should be a more correct solution that doesn't force end users to use a bundler.

[astrojuanlu]

After reading around a bit, I think myst-to-md (and possibly the other packages in this monorepo) could make some adjustments to ease things for end users.

This is how package.json looks like at the moment:

https://github.com/executablebooks/mystjs/blob/5c83714ee2239278c7a30c873832c2967a5c0e85/packages/myst-to-md/package.json#L8-L20

  "main": "dist/cjs/index.js",
  "module": "dist/esm/index.js",
  // ...
  "exports": {
    ".": {
      "require": "./dist/cjs/index.js",
      "import": "./dist/esm/index.js"
    }
  },

However, I note that:

• It uses the module key, which was a 2016-2017 proposal (a, b) seemingly used by Webpack but that never got officially implemented in Node.js, being replaced instead by the exports proposal in 2019.
• It does not have a type key, which means that it has the default value "commonjs", and as such all .js files are treated as CommonJS modules.
• All the exports have .js extension.

This is probably the reason why I'm getting a warning (and then a SyntaxError) when trying to import from myst-to-md, even though my own package.json contains a "type": "module" (hinting that the problem is in the dependency itself).

What the TypeScript docs do instead is using a specific extension to separate CommonJS from ESM:

// package.json
{
  "name": "my-package",
  "type": "module",
  "exports": {
    ".": {
      // Entry-point for `import "my-package"` in ESM
      "import": "./esm/index.js",

      // Entry-point for `require("my-package") in CJS
      "require": "./commonjs/index.cjs",
    },
  },

  // CJS fall-back for older versions of Node.js
  "main": "./commonjs/index.cjs",
}

Therefore, even though I have little experience with the Node.js ecosystem, I think there are essentially 3 ways forward:

• The current way (not touch anything). The default experience is suboptimal, but seemingly bundlers cope with it and make everything work.
• Keep the default type (hence CommonJS), but use the .mjs for the ESM exports (module and exports.require).
• Change the type to "module" (hence ESM), and use .cjs for the CommonJS exports (main and exports.import).

I'd send a pull request myself but I considered it would be good to discuss it here first. Thoughts?

[rowanc1]

I would certainly appreciate a PR, we are certainly in a local minimum and I know we are doing some of these things sub optimally. Appreciate the research you have done. :)

One of the most difficult things that we have had problems with is getting Jest to work, which is the testing framework we are using. Those are both very slow (jest goes through and compiles a lot, and doesn't handle esm + typescript very well), and it can be pretty finicky and has a different (and dated) opinion on what is "right". Introducing unified (which is esm only) in ~September 2022, meant that we had to put a bunch of work to make things work again, that was the last time we touched packaging in any serious way.

I think another thing that will need to happen to change to a module is changing all of the imports internally to add file extensions. @fwkoch did an attempt at this in January, but we abandoned it at that time because we couldn't get the test-runners back up and running. :(

One of the pieces of advice I have seen since then is to compile/bundle each library using esbuild, and then use that dist file in the test-runners. We haven't tried that yet, and it might be a way for us to migrate (while keeping the tests running!).

Getting this setup, improved and working in all the various scenarios on one of the packages would be soo helpful as a template to role out to the rest of the packages. I think that it can be done one at a time as well.

[astrojuanlu]

Naturally I had lots of unknown unknowns :) Maybe the option of using the .mjs extension for the ESM exports fixes this use case while minimizing the amount of changes needed elsewhere. I'll keep investigating how can I have a Dev environment, test things locally etc before hammering your CI and hopefully contribute a PR soon.

[astrojuanlu]

Oh wow, the story of TypeScript and file extensions is... intense microsoft/TypeScript#18442 (people trying to fix this problem with some convoluted renaming scripts)

Long story short, "TypeScript by itself won't ever support dual target".

This comment does contain some practical advice (emphasis mine)

That's a shame. In that case can you offer any suggestion for publishing dual packages from single codebase?

And, to further elaborate: I'd avoid it, if it all possible. Probably just ship cjs unless you have a good reason not to. If you wanna expose an esm API, then, ship a cjs core with a handwritten esm wrapper, rather than duplicating all your shipped implementation code. If you wanna do something that only works in esm (eg, top level await) but still provide something for cjs users, then you'd write an esm core with a janky async cjs wrapper (and only if you begrudge your cjs consumers and want to watch the world burn). A "dual mode" package that ships transpiled copies of the same input code is a massive antipattern and the node docs highlight some downsides to that approach (singleton hazards, code duplication within runtime). You can semi-justify it when providing a version for pre-esm (so node 11 and older...) node - but at that point you're shipping cjs still, so you should just only ship cjs, imo.

It looks like I might have arrived ~1 year too early after all 😅 Okay, I give up - will look into esbuild next.

[rowanc1]

Making some progress on moving to ESM only @astrojuanlu!

jupyter-book/mystmd#417

I moved from jest --> vitest, which allowed us to sidestep all of the nonsense related to jest and cjs. I think this will help things out, but might be a bit bumpy as we figure out some downstream packaging issues.

[astrojuanlu]

Music to my ears 💗 I'll give this a fresh look soon. Thanks a lot!