Async API?

[mpizenberg]

I've been playing with making a WebAssembly solver for the Elm ecosystem, with the aim of it being easily usable for tooling written in JavaScript. The wasm API I've settled on for the time being looks like this:

let wasm = require("elm-solve-deps-wasm");
let solution = wasm.solve_deps(
  rootPackage,
  fetchDependencies, // function
  listVersions // function
);

It works fine but requires that the fetchDependencies and listVersions functions passed as argument are sync. However, in the NodeJS ecosystem, reading files and making http requests is mainly done with async. So this API forces the user to use sync versions of file IO, which is fine, and sync versions of http requests which is a bit more annoying to do.

In order to be able to write those two functions with async, it means we also need an async version of the DependencyProvider trait, where getDependencies and choosePackageVersion would return futures instead of normal values. I'm not sure if it's easily done, because async and traits are quite painful together from some of my readings, but I wanted to raise this use case.

[Eh2406]

Thank you for giving this a try! I look forward to reading over your elm example! It is related to pubgrub-rs/advanced_dependency_providers#6.

The first problem is that we don't fit well into the "just create futures and process them when they're done" paradigm. While listVersions is off doing network stuff, it's entirely possible that unit_propagation has discovered the parent cannot be used due to an unrelated conflict.
We also don't fit well in the underlying "cooperative multi-threading" paradigm. unit_propagation is synchronous computational work that is sometimes fast enough to do on the main thread, and other times without warning very blocking. That makes it very uncooperative with respect to "cooperative multi-threading".
However it is a real user need, so whatever "the best we can do" is we need to implement or document it.

What I had in mind was:

• an async DependencyProvider has a cache of package -> promise<listVersions result>.
• choosePackageVersion looks over the packages and asynchronously queues any items not in the cash. (Which will get processed by some other thread.)
• choosePackageVersion it then filters too only items whose result is available and pics one.
• if none are available then it blocks for one of them to be available.

Of course this may fall apart when we actually try and implement it.

[Eh2406]

CC #138

[konstin]

In uv, we moved to running pubgrub in a separate, dedicated thread (astral-sh/uv#3627) and the single threaded tokio runtime as main thread (astral-sh/uv#4934), which are faster for us.