HLS dynamic plugins loading proposal

[KovalevDima]

Is your enhancement request related to a problem? Please describe.

I’m often frustrated with the HLS plugin architecture for the following reasons:

1. It forces me to rebuild HLS every time I want to modify its plugin set.
2. It encourages me to override default package options to remove any plugin dependencies.
3. It restricts me from implementing my own plugins by requiring them to be pushed to the HLS monorepo.
4. It implies that everyone must compile a lot of dependencies with default plugin set

Describe the solution you'd like

In the cabal.project file, we can specify a list of packages that should be compiled and loaded into GHCi. For example:

extra-packages:
  hls-fourmolu-plugin
  hls-stylish-haskell-plugin
  hls-stan-plugin
  hls-floskell-plugin

AFAIK, HLS loads a GHCi session and has access to all of the project dependencies. If we modify the hls-plugin-api to:

1. Support discovering available plugins
2. Loading compiled plugins from GHCi session

then we will have a simple dynamic plugin loader that provides users with the following capabilities:

1. Defining their own plugins for local projects without modifying or redistributing HLS.

   -- cabal.project
   packages:
     ./myAwesomeProject
     ./hls-myAwesomeProject-plugin

2. Using all the HLS plugins available on Hackage without needing to recompile HLS.
3. Overriding any HLS plugins without recompiling HLS.

As a result, we will have a much more modular HLS architecture, which leaves the support of experimental or controversial plugins to the maintainers. It will also provide the ability to experiment with HLS with less effort. Additionally, it will simplify HLS distribution by making it independent from at least some of the existing plugins, reducing the heavy dependency footprint of HLS

Describe alternatives you've considered

I've been thinking about HLS recompilation before start. But I think it will increase HLS start time a lot

Additional context

Inspired by discussion #4464
I don't know if it is possible to implement, so please leave any feedback about this proposal

[michaelpj]

I thought we had some discussion of this before, but I can't find it. So I'll write out my thoughts again.

The biggest problem is that I don't see any way of doing this that doesn't have terrible costs. I don't think your proposed solution works (sorry), but at least in principle dynamically loading Haskell library object files at runtime can be done. However, it enormously complicates distributing and building HLS, which is already by far the worst part of HLS development and which sucks up far more of our cycles than I would like.

I think there is some extrinsic evidence that this is hard: I know of no tool with plugins that work by distributing object files. AFAIK they all work through either having their own simple programming languages that are distributed as source (elisp, vimscript, etc.) or by distributing a relatively platform-agnostic intermediate format with good runtime support for loading them (JVM classfiles, etc.).

So that's the cost side. I also don't think the benefits are that great.

• Having all the plugins in a monorepo significantly simplifies development, and has not as far as I know prevented anyone keen from writing a plugin.
• There is not great demand to write HLS plugins, partly because they're just not that easy to write.
• The vast majority of HLS plugins represent logical chunks of functionality, not things you might want to enable or not. That is, we use plugins more as a code organization scheme than as a means of conditionally enabling functionality.
• I don't see that we would gain any modularity that we don't already have by organizing our code in this way.

So it seems to me that:

• The costs are high
• The benefits are low
• There is already a viable workaround for people who want to trim the dependencies: the cabal flags.