Resolving module's name within node_modules

[theseanl]

I'm trying to use tsickle to write externs from type declarations from DefinitelyTyped packages.

Although typescript does not transpile *.ts files in node_modules/@types by default, by manually providing them to the compiler, it was possible to trigger tsickle to generate externs for them.

However, for type packages that are using module name with relative paths (e.g. lodash), tsickle generates broken externs, because module names are resolved with resolveModuleName function and the resolved path is in a node_modules directory relative to the current typescript project root. Then tsickle does not resolve such names and writes externs on inconsistent namespaces.

Would there be any way to properly generate externs in such a use case?

It would have worked if resolveModuleName have discarded resolved path iff it is piercing node_modules package boundary, i.e.

• For a file, its package boundary is its nearest ancestor node_module directory's child directory.
• Discard resolved and return imported literally only if a package boundary of pathOfImportingFile and resolvedModule is different.

and it looks like to be a behavior that was intended from the beginning, if such a change would not break other projects using tsickle, would it be possible to adopt such changes?

[evmar]

externs are always global (in TS lingo: scripts, not modules). But typings can be modules, containing import/export statements as you observe here. We don't yet have a good plan for how to reconcile these together. Currently if you give tsickle a d.ts that is a module, it puts all of the types into a fake namespace that Closure doesn't even see, so I am not sure it helps you too much.

We'd love your help on this, but first I'd like to understand your plan for how this works, independently of node_modules.

[theseanl]

In my use case, that namespace being fake is not a problem; it needs to be consistent across different d.ts files that augments the same module declaration. Currently it is not consistent and the generated externs causes Closure Compiler error.

I am currently grabbing the fake namespace (which is predictable) to use as a type declaration of global variable, and replacing imports of such modules to a reference to a global variable, similar to how one provides an external imported module as a global variable by rollup's outputOption.globals option. This is how it works in a tsickle wrapper I'm working on — It will definitely help me a lot!

[evmar]

(Wow, tscc looks neat! I would love it if you could tell us how tsickle ought to work to make tscc easier. For example, maybe we should just remove the tsickle binary (main.ts) entirely, and instead say it's only a library and that if you wanna run it yourself you should use tscc? I would warn you that we make breaking API changes frequently though.

I am confused by
https://github.com/theseanl/tscc/blob/master/packages/tscc/src/transformer/externalModuleTransformer.ts#L99
because we don't special-case tslib in the google3 setup as far as I know. Anyway, maybe we candiscuss this on another bug.)

Can you describe the inconsistent namespace thing in more detail then? I guess I don't follow how node_modules is specifically involved, relative to the more general issue of "we don't handle module typings".

[theseanl]

I created a demo for the issue - https://github.com/theseanl/ts-declare-module-test

off topic

Thanks for your words! Currently there were something that weren’t possible with public APIs, like blacklisting types for external modules. In tscc setup, if a module react is declared as an external module,

import * as React from ‘react’

should not produce something like

goog.requireType(“node_modules.react.umd.react_production_min”)

so I am hacking isBlacklisted function exported from type_translator.ts. This is produced by jsDocTransformer and I’m not sure if I can use some source transformer to remove the import before jsDocTransformer and keep the source file to be a valid typescript. Would there be a way to provide such a functionality to blacklist type by ts.Symbol.name as a public API?

Regarding tslib(I have no idea about what is "google3 setup") this is due to the different pathToModuleName function it feeds to the tsickleHost, it actually resolves the module name and replaces it with its path. So if a consumer's package have tslib as a dependency, any require('tslib') will be replaced with something like goog.require('node_modules.tslib.index') which is undesirable because the custom tslib coming with tsickle declares goog.module('tslib'). Maybe this is undesirable and I should not use googmoduletransformer and feed es6 module to compiler directly.

[evmar]

I think there are maybe two problems here:

1. we do something wrong with module augmentation
2. that is complicated further by some weird special casing of node_modules in the code you linked

So I think we here should maybe figure out module augmentation in the non-node_modules case (like your example) and the come back to the larger issue of what to do about imports from paths like 'react'. The latter touches on some much larger problems with how to make libraries like react work in closure compiler at all. (If you're a googler, read go/tpl-js.)

[theseanl]

I thought module augmentation was working well in general, the only problem here was the handling of node_modules. I'm not sure what you are talking about the point 1. My example shows problem only when node_modules are involved. So here IMO only the node_modules case needs to be fixed and the larger problem is less relevant.

To fix an immediate issue of using lodash, I went ahead with an ugly solution here, IMO tsickle may drop the special casing of node_modules or be more sensible in it.

Regarding the general issue, I'm not a googler so I have no idea about how external library is used with closure compiler internally, but many module bundlers like rollup and webpack have a way of marking certain module as external, so I guess such a functionality may be added to closure compiler, or tsickle creates externs for a hypothetical exports object which user provided a global name somehow via an API.

[evmar]

I wrote a longer thing above for you, let me know if it helps.

I believe the problematic resolveModuleName code you wrote was intended for the case where someone runs tsickle first and then runs the output through something other than Closure, which seems like madness to me. I think Angular did this for a while for reasons I don't fully understand. I think I agree that tsickle should drop this special-casing since (as I describe in #1041) it really just doesn't work.

I don't really understand your solution, do you run the closure output through rollup afterwards?

[theseanl]

I believe the problematic resolveModuleName code you wrote was intended for the case where someone runs tsickle first and then runs the output through something other than Closure, which seems like madness to me.

In reality, I'm passing the generated code directly to Closure.

I don't really understand your solution, do you run the closure output through rollup afterwards?

No, rollup is a separate, alternative compilation means. Closure output is the final output.

It's unclear to me what is unclear about my use case to you; In the current situation, when tsickle is run on https://github.com/theseanl/ts-declare-module-test, the output externs.js file cannot be fed to Closure, it will produce errors about undeclared namespaces. If the resolveModuleName function body is replaced to what I wrote in above commit, externs will get correct namespaces and Closure will consume it happily.

My idea have been to make the option 2 in your explanation #1041 (Thanks for writing it) systematic:

1. Users write import React from 'react', in order to get Typescript typings.
2. The build tool configures transformerHost to do process type declaration files of module 'react' that Typescript located to us -- usually in node_modules directory.
3. With the help of some custom TS transformers, the tool removes import React from 'react', and replace references of React to a user-provided global variable name. Doing it requires suppressing goog.requireType('react') and substituting const React = require('react') to something like const React_1 = React.
4. To inform Closure about such a user-provided global variable name, the build tool generates additional externs for such a global variable, like

   /**
    * @const
    * @type {some$file$name$mangled$by$tsickle}
    */
   var React = {};

   tsickle writes module-scoped externs to certain mangled namespace like this, so I am grabbing that namespace to create externs like this. To my understanding, this will provide the required information to Closure, please correct me if I'm wrong.

As I understand, the goal of tsickle is to produce codes that are at least consumable by Closure. The current state is that when it is fed with certain files spread across node_modules boundary, the output extern will not be consumable by Closure. That's why I expect this issue to be fixed from the tsickle's side.

[evmar]

@alexeagle I think is the author of the node_modules special case, maybe he can say whether it's safe to remove?