Deno-like Execution Environment and Ecosystem

[zicklag]

I just had an idea that I wanted to get written down for the HVM.

It'd be cool to have an HVM equivalent to Deno. A couple specific points from Deno that would be cool:

• Capabilities/permissions setup like Deno to control filesystem/network access.
• Lack of a "package manager" by using URLs or relative paths to import modules

And a couple other ideas that would be interesting:

• We come up with a linking standard for HVM languages.
  • This would be important for having HVM languages be able to inter-op with each-other.
  • The lack of a linking standard is actually the single largest barrier for me using WASM to create highly efficient video-game mods for WASM languages.
  • This wouldn't allow any HVM language to seamlessly inter-op, because the might each have different type systems, so they would have to have some sort of extern types, but that's still doable.
• We could have the HVM Deno-equivalent automatically compile any HVM language that has it's compiler compiled to HVM.
  • So for instance, if you included a HOP file in your program, which has it's compiler written in HVM, that file could say that it must be compiled using https://url/to/hop.hvm before being executed.
  • The runtime would download hop.hvm, and run it on the hop source file, to produce an HVM file, that would then be linked to the other modules.
  • For any compilers that can't be run on HVM ( like I'm not sure if Kind can because it's written in Rust ), you'd just have to compile the module to HVM before including it, similar to how you use TypeScript with Node.js.

[Boscop]

You raise a lot of interesting points that all relate to each other: Sandboxing, effects, "ABI" for interop between HVM programs. (These also relate to FFI, both in terms of having a stable interface between a native host and HVM runtime, and HVM runtime calling into native libs, but also the sandboxing/permissions related to the FFI boundary.)

I've been thinking about the same things, sandboxing with fine-grained permissions would make a lot of sense for the HVM.
Previously I was using different languages to extend a host's functionality via extensions, e.g. Gluon language for scripting a Rust host, and another application was an online playground (code had to be executed server-side because the Gluon interpreter couldn't be compiled for the wasm target)..
Sandboxing is increasingly important for security, and e.g. wasm runtimes are built with that in mind.
There's an interesting discussion here: helix-editor/helix#3806
The Helix editor authors were discussing different embeddable scripting languages for the upcoming extension system, sandboxing was very important to that discussion.
And I do think the ideal embeddable scripting language that would be a better choice than wasm and lua and lisp for these use cases would be a fast functional runtime with sandboxing. The HVM could become a prime candidate for these use cases.

Regarding sandboxing in functional languages:
For example the Roc language has the concept of platforms which provide different effects, and an application author can choose which platform to run on.
https://www.roc-lang.org:

Roc aims to be a purely functional programming language. This means all Roc functions are pure functions, and all effects are managed effects instead of side effects.
A major motivating reason for this is to facilitate tooling. For example, in the future the goal is that Roc's test runner won't bother re-running tests whose outcomes could not possibly have changed (because they were pure functions whose inputs did not change). Tests that contain only pure functions can be trivially run in parallel, and they will never flake. Additionally, having the guarantee that the application contains only pure functions can also make certain debugging tools more reliable, such as time travel and retroactive tracing.
Roc also takes a novel approach to managed effects. In most programming languages, the standard library contains both data structures and I/O primitives (e.g. for using the file system or the network), and then you might decide to use a framework on top of that standard library.
In Roc, every application is built on a platform. A platform is like a framework except that it also provides I/O primitives and behind-the-scenes memory management. (Roc's standard library only contains data structures.) In practice, this means that using Roc feels similar to using any other programming language where you've chosen to use a framework, except that the documentation for your I/O primitives comes from the framework instead of the standard library.
This might sound like a minor distinction, but it turns out there are a lot of surprising benefits to organizing things this way, which would be impossible to achieve without having platforms as a first-class language concept. The Edges of Cutting-Edge Languages goes into more detail about some of these benefits.

Also interesting talk: A Taste of Roc — Richard Feldman: https://www.youtube.com/watch?v=6qzWm_eoUXM

---

We should probably think about how to model these effects in the HVM, and how it affects the design of the FFI when calling into native/unsafe libs. (The permission to call into native libs could be coarse-grained/boolean or more fine-grained, depending on which metadata we'll have about the effects of the lib's functions, e.g. we could try to distinguish between network and file-system access across the FFI boundary..)

---

Lack of a "package manager" by using URLs or relative paths to import modules

That's a different aspect entirely. I think it makes sense to have a manifest for version management of deps (and you'd have a manifest file anyway to specify metadata of your HVM package). Instead of having to replace a URL in N source files (and potentially overlooking some), you just have to update the version in the manifest file. Also it would be easier to override deps (e.g. temporarily using your fork that contains a fix with [patch] in Cargo.toml), or specifying which registry should have precedence when pulling in deps, this is important for many companies who have their own internal package registry which contains their forks of packages that also exist in the default registry, so their internal package registry takes precedence.
If Rust packages didn't have a Cargo.toml manifest, it would be a lot more hassle to manage your dependencies, and it would still require to actually have cargo resolve those dependencies according to semver, and build them etc.
Allowing imports via relative paths in addition to having a dep manifest makes sense when importing files within the same workspace.
Also e.g. if you're writing extensions for your host in HVM and they import some local files that aren't published as a package anywhere. (Also when HVM as a lib will have a customizable prelude, the host can already include certain symbol definitions for embedding use cases.)

[glyh]

I don't think Url for dependency management is a good idea. go use github(which is sorta like URLs) to manage dependencies and it becomes one of the points people keep criticizing it: if you fork a library, you'd have to update the imports in the source code. That wouldn't be ideal at least.

[zicklag]

I think using URLs is only good if you have the options to combine with import maps, which makes them not much different than a normal package.json or Cargo.toml kind of workflow, but still allows you the flexibility of not needing an explicit package manager in the same way.

Still, I'm not sure that it's ultimately the best option either.