Allow Mix.install/2 to run several times in the same instance

[josevalim]

The process should be roughly like:

1. Create a new directory for the new installation
2. Run mix deps.get
3. Compare the versions and see which entries were added or removed
4. Run a fixpoint computation until we find all application that depends on something that was added or removed
5. Stop all apps, unload all modules, and remove all code paths for previous dependencies + consolidated protocols
6. Copy the _build and deps entries for the ones that did not change
7. Continue as usual

/cc @wojtekmach

[thbar]

Coming from:

• Incremental compilation? livebook-dev/livebook#2303

Something increasingly useful I'm doing for prototyping of feature is: use the trick described in https://github.com/livebook-dev/livebook/blob/main/lib/livebook/notebook/learn/intro_to_livebook.livemd#elixir-integration-and-use-cases to re-use the main app config & dependencies inside an otherwise standalone "exs" script, but add a couple extra dependencies in the same Mix.install/2 call for prototyping purposes.

This is very useful to give birth to complicated features that will require all the app context, but delay the inclusion of dependencies until the last required moment, something that is cleaner in terms of maintenance IMO.

Having incremental compilation here would be especially useful, because we otherwise recompile the "whole app" each time we add one dependency on the script calling Mix.install/2 directly.

Is the current issue about that, or something bigger? Thank you!

[lukaszsamson]

This is something that would be beneficial for language servers as well. ElixirLS already has custom (and most certainly buggy) implementation of steps 2-5

[josevalim]

@thbar this issue is about that, yeah. :)

@lukaszsamson I am not sure how applicable this is outside of Mix.install. The issue with Mix.install is that it compiles everything from scratch on each new dependency and the main goal of this task is to minimize that.

For an actual project, which Elixir LS is running on, you are not compiling everything from scratch, so the benefits are minimal. Plus, in all measurements I have performed, stopping all applications, removing all modules, and pruning all code paths was always slower (and bug proner) than restarting the app. For Elixir LS, I would consider isolating the app runtime from the language server itself, so you have more flexibility of when to restart the runtime.

It may be that we use a similar solution for Mix.install/2. Instead of allowing it to run multiple times, we can simply allow a previous install directory to be used as a reference (and then still restart the runtime).

[josevalim]

#13378 implements a simpler version of this that will allow us to test how easy deps are to migrate.

[viniciusmuller]

Hey, I would like to work on this feature :)
Would it be possible to get a simple overview on the current state and what's still needed?

[josevalim]

Hi @viniciusmuller ! I believe the description is up to date. The previous PR is helping us battle test if copying apps from one folder to the other works, and it seems it does indeed, but the bulk of the issue here is yet to be implemented.

[josevalim]

Btw, one approach to tackle this is to only support adding new dependencies. But we need to be careful with optional dependencies, which may cause previously defined deps to be recompiled. We may need to crash in those cases and ask to try from scratch.