Further Enhancements to the Module System

[leeyi45]

Notable Changes

export from resolution

When using export from declarations, the system will now resolve the symbol through the directives to where the symbol was defined:

// a.js
export const a = "a";

// b.js
export { a as b } from './a.js';

// c.js
import { b as c } from './c.js'
// gets translated to
import { a as c } from './a.js'

This change may cause snapshots that use import directives to produce different import statements when preprocessed.
Should fix #1416 by introducing support for such export declarations

Asynchronous Imports

Addresses #1348 partially by introducing a new set of functions that use fetch to retrieve module files. However, the interpreter does not support asynchronous operation, so the original synchronous module loading code has been left in.

Import Preprocessor

The import preprocessor has been changed to treat local and source imports more uniformly:

• The preprocessor now supports resolving and checking for undefined symbols for local modules as well
• Though export * statements are not currently supported by other language features, the preprocessor has the ability to detect reexported symbols should such statements become available in the future
• Options for module resolution have been added, allowing specifications like import a from './a' to be resolved to import a from './a/index.js', similar to how Node and browsers resolve modules. This should help work toward the js-slang repl being able to execute multiple files

Assertions

Introduce a newAssertionError type that derives from RuntimeSourceError to gracefully handle unexpected conditions that arise during program execution, along side an assert function. Using a custom function allows us to modify the errors assertions throw, and removes the need for polyfills for the NodeJS assert module to work with the browser.

Minor Changes

Execution Options

• As part of an effort to make debugging js-slang easier, several options to display debugging information have been added to execution such as logTranspilerOutput have been added
• Options controlling how imports are resolved have also been added. For example, js-slang can be configured to avoid loading tabs, which would come in handy especially for execution contexts that can't display tabs

Refactoring of AST utils

• Code relating to manipulating an AST have been collected under src/utils/ast. Where possible, type guards that were spread out across different parts of js-slang have been consolidated.
• import * as es from './utils/ast/types' can be used to replace import * as es from 'estree'. This will provide extra AST node types and reduce the amount of type checking needed when dealing with an AST. All types are compatible with estree.

[coveralls]

Pull Request Test Coverage Report for Build 5406352152

• 711 of 831 (85.56%) changed or added relevant lines in 50 files are covered.
• 7 unchanged lines in 3 files lost coverage.
• Overall coverage decreased (-0.02%) to 83.452%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/modules/preprocessor/index.ts	97	98	98.98%
src/modules/preprocessor/transformers/removeImportsAndExports.ts	11	12	91.67%
src/utils/ast/typeGuards.ts	16	17	94.12%
src/interpreter/interpreter.ts	9	11	81.82%
src/stdlib/list.ts	1	3	33.33%
src/utils/arrayMap.ts	6	8	75.0%
src/modules/moduleLoader.ts	13	16	81.25%
src/runner/sourceRunner.ts	16	19	84.21%
src/transpiler/transpiler.ts	37	40	92.5%
src/modules/preprocessor/transformers/hoistAndMergeImports.ts	34	38	89.47%

Files with Coverage Reduction	New Missed Lines	%
src/ec-evaluator/interpreter.ts	1	91.34%
src/name-extractor/index.ts	2	65.7%
src/index.ts	4	55.97%

Totals
Change from base Build 5387187177:	-0.02%
Covered Lines:	10852
Relevant Lines:	12603

---

💛 - Coveralls

[leeyi45]

I have since taken several of @ianyong's suggestions into account and modified this PR appropriately, the last remaining issue is to solve the error that comes up when running jest. Then I will be able to move forward with unit testing.

[leeyi45]

As a comment for the EC-Evaluator team @dickongwd @vansh284, would there be an issue if the Program command evaluator were written as part of the ECEvaluate function? Programs should not contain other programs as children if I understand correctly, so there's no need to write it as a separate evaluator:

function ECEvaluate(...) {
  for (const node of program.body) {
    // Evaluate imports first
  }
  for (const node of program.body) {
    // Run command evaluators
  }  
}

This makes it easier to pass options to the the code that handles module imports, and also removes the need to make every evaluator async.

[vansh284]

As a comment for the EC-Evaluator team @dickongwd @vansh284, would there be an issue if the Program command evaluator were written as part of the ECEvaluate function? Programs should not contain other programs as children if I understand correctly, so there's no need to write it as a separate evaluator:

function ECEvaluate(...) {
  for (const node of program.body) {
    // Evaluate imports first
  }
  for (const node of program.body) {
    // Run command evaluators
  }  
}

This makes it easier to pass options to the the code that handles module imports, and also removes the need to make every evaluator async.

You are right in saying that Programs wouldn't contain other programs as children, so the final evaluation would still be the same.

However, (I'm not sure if you are aware) the 'agenda' in the ec-evaluator is visualised in the frontend as part of the Env Visualizer. The command evaluators act on each of the 'agenda' items, so if we have no command evaluator for the Program, it cannot be pushed on the 'agenda'.

So if instead of having a Program command evaluator, we directly evaluate the program at the start, the students wouldn't be able to see the first step of the visualization (which is the Program item on the agenda) as it would be skipped.

Maybe you can let us know exactly what your goal is, so we can think of a way to make it work with the ec-evaluator.

[leeyi45]

As a comment for the EC-Evaluator team @dickongwd @vansh284, would there be an issue if the Program command evaluator were written as part of the ECEvaluate function? Programs should not contain other programs as children if I understand correctly, so there's no need to write it as a separate evaluator:

function ECEvaluate(...) {
  for (const node of program.body) {
    // Evaluate imports first
  }
  for (const node of program.body) {
    // Run command evaluators
  }  
}

This makes it easier to pass options to the the code that handles module imports, and also removes the need to make every evaluator async.

You are right in saying that Programs wouldn't contain other programs as children, so the final evaluation would still be the same.

However, (I'm not sure if you are aware) the 'agenda' in the ec-evaluator is visualised in the frontend as part of the Env Visualizer. The command evaluators act on each of the 'agenda' items, so if we have no command evaluator for the Program, it cannot be pushed on the 'agenda'.

So if instead of having a Program command evaluator, we directly evaluate the program at the start, the students wouldn't be able to see the first step of the visualization (which is the Program item on the agenda) as it would be skipped.

Maybe you can let us know exactly what your goal is, so we can think of a way to make it work with the ec-evaluator.

Perhaps I could refer to the implementation within the stepper:

async function evaluateImports(
  program: es.Program,
  context: Context,
  { loadTabs, wrapModules }: ImportTransformOptions
) {
  let importNodes: es.ImportDeclaration[]
  ;[importNodes, program.body] = partition(program.body, importDeclarationFilter)

  try {
    const importNodeMap = arrayMapFrom(
      importNodes.map(node => {
        const moduleName = node.source.value
        assert(
          typeof moduleName === 'string',
          `ImportDeclarations should have string sources, got ${moduleName}`
        )
        return [moduleName, node]
      })
    )

    const environment = currentEnvironment(context)
    await importNodeMap.forEachAsync(async (moduleName, nodes) => {
      await initModuleContextAsync(moduleName, context, loadTabs, nodes[0])
      const functions = await loadModuleBundleAsync(moduleName, context, wrapModules, nodes[0])

      for (const node of nodes) {
        for (const spec of node.specifiers) {
          let importedName: string
          switch (spec.type) {
            case 'ImportSpecifier': {
              importedName = spec.imported.name
              break
            }
            case 'ImportDefaultSpecifier': {
              importedName = 'default'
              break
            }
            case 'ImportNamespaceSpecifier': {
              throw new Error('Namespace imports are not supported!')
            }
          }

          declareIdentifier(context, spec.local.name, node, environment)
          defineVariable(context, spec.local.name, functions[importedName], true, node)
        }
      }
    })
  } catch (error) {
    // console.log(error)
    handleRuntimeError(context, error)
  }
}

This is directly called by getEvaluationSteps:

// the context here is for builtins
export async function getEvaluationSteps(
  program: es.Program,
  context: Context,
  stepLimit: number | undefined,
  importOptions: ImportTransformOptions
): Promise<[es.Program, string[][], string][]> {
  const steps: [es.Program, string[][], string][] = []
  try {
    const limit = stepLimit === undefined ? 1000 : stepLimit % 2 === 0 ? stepLimit : stepLimit + 1
    await evaluateImports(program, context, importOptions)
    // starts with substituting predefined constants
    // etc...

I am able to access the loadTabs and wrapModules options. Notice that evaluateImports is also an async function. For the EC-Evaluator to be able to make use of async module loading code, the code that evaluates Program nodes needs to also be async.

[leeyi45]

Due to the circumstances, I was unable to work on this. Given the large number of changes that have been merged since I will reopen this with a fresh branch