module loader as browser feature

[sashafirsov]

Module loader is a concept of trusted web application assembly from locally and CDN cached of transformation pipelined modules.

Once implemented it would allow to

• manage web application modules mapping by web application descriptor (WAD) allowing only modules approved by web app owner
• module types mapping. I.e. js(x), md(x), css/scss, svg, ts(x),... to app-defined transformed module format
• defines transformation pipeline
• defines protocols of source and transformed module retrieval. NPM/SCM/git repo | CDN | local cache.
• defines protocol for triggering the transformation pipeline and results retrieval( to be cached in local registry or on CDN ). Transformed modules are signed by source+transformation descriptor private key, public key exposed for validation.
• keeps cross-domain cache registry of transformed modules in browser. The sources identified by e-signature (even from different domains) would be matched to transformed module in cache and cached value re-used.
• semver updates of registry with fine-grain control by web app descriptor
• the registry be available on
  • browser
  • server(like NodeJs)
  • CDN + compute CDN( like CloudFlare workers )
  • CMS

Transformation results

the primary goal is delivery of transformed module sufficient to be used by web app. But different platforms could go further and do some more optimization after common format delivered.

• more platform-specific optimization. Like binary AST by Spydermonkey, locally compressed binary XML, etc.
• legacy support. Extra transformation to ES5, adding polyfills.

In order to decide whether application uses trusted sources and code has sufficient quality, complimentary reports could be triggered during "release" process. Those reports are stored next to cached source and transformed results and allow to have a complete in-deps review of all engaged modules.

unit test

load/performance test

coverage report

code quality scan

vulnerability report

Dependencies

The dependency tree is created for all direct and indirect used modules. As each sub-module would have a sufficient identifiable source and transformation info, complete dependency tree would be available with whole report set ^^ (unlike current build toolchains like WebPack which produce coverage just for owned sources ).

Benefits for targeted audience

DX

Target: sub-second compile time.

Local development does not need full transformation and all reports. Instead it would streamline for sub-second app compile+loading speed. Most of popular libraries would be available over CDN as source and binary modules with sourcemaps. There is no need for bundling:

• libs dependencies would be cached in browser once retrieved from CDN. In DEV environment semver dependencies would be cached forever on client(browser).
• own module cache identified by hash and would need a recompilation only when changed.
• transparent build would be executed on load by browser time or offline as locally hosted CDN. To be clear, the compilation could run as within browser as on local host. In case of in-browser the build toolchain would be loaded from same cache. Mozilla has clear speed advantage there as pre-compiled AST is a module presentation. On server side it is recommended to use shared FS with OS cache sufficient to reuse modules cache from memory. Docker image with functionality of semver CDN and modules registry for local host would be provided to be customized according project needs. Same image could be used as part of CI/CD.

Auto-reload of web application on module source change

and following transformation would be integrated into loader.
The local FS monitor paired with web app loader would track the dependencies changes.

Browser app loading and upgrade

Smoother than ever. Cross-domain cache of semver published modules would eliminate the load time for most of modules for popular libs and previous builds of your web application. Only updated modules would be re-loaded.

Application loader would use WAD for locating dependencies, local cache first, CDN(s) rating would be computed on 1st load and fastest be used as primary, fallback on missing/outdated modules to another CDN(s) up to local build environment if binaries are missing.

Since most of libs are reused across many domains and apps, their modules would be picked from domain-neutral local cache. For security reasons each module loading would validate the module integrity even if it is loaded locally. Validation is not a heavy process as checksum calculation would be going along with loading stream. When/If the proposal would become a part of browser stack, each part would be natively optimized. meanwhile the initial draft would have polyfills, plugins, customized build for browser( most likely Mozilla ).

CI/CD

The modules deployment itself is not a subject for CI/CD. That would be covered by environment ( DEV, QA... PROD) CDN/ module registry.

CI/CD would be creating the set of reports for the build (pr/branch, etc.), publishing them on environment CDN/module registry next to already compiled sources/binaries. As most of modules of particular version would be covered by previous report, CI/CD would just process uncovered modules and create summary report which would be published as part of WAD. Which would reduce deployment and reports generation time by level of magnitude for enterprise scale app ( most of modules would be unchanged and reports would be reused )

Security/trust validation

Is not a subject for the web app vendor. Those could go lazy and take a shortcuts compromising the consumer safety. As of now consumer can check as little as valid https certificates exposed by browser if one of 3rd party scripts host is not well maintained. There is no way to validate whether parts of app actually made by secure and trusted sources. Transparency of web apps would give a consumer an option to use native app instead of vulnerable web app or go physically to the vendor. The licensing of business from security prospective could be changed to particular release of web application.

Due to semver support the security fixes would be propagated immediately as the fix is published. Web app owners could require the security review seal from vendor(s) in order to pick the update. Security reviewer would be able to validate all sources including dependencies for vulnerability scans and seals by trusted verifier.

[sashafirsov]

Interesting twist on dependency tree management. Most of modules even if changed over time keep much if not most of code across releases. Unfortunately due to publishing as a bundled binaries not builder not browser could reuse unchanged parts.

In epa-build the intermediate result would be kept as AST which in theory allows to detect the parts of code used in app dependency tree and difference between releases in libs. Resulting in dups elimination.

That would be one more layer of optimization - app specific dependencies "tree shaking" which is more powerful of usual as allow to work cross-release of app and dependencies.

[sashafirsov]

The storage for in-browser module registry candidates:

• indexedDB
• Virtual File System ( chrome+edge only )
• CacheStorage
• localStorage