[proposal] Make elastic-package an extensible CLI

[marc-gr]

Previous context: #479
Experiment validating the approach: #1416

Context

Elastic-package provides some functionality that is general to all packages around testing, validation, etc.

There are other operations that might be less general, for example ECS updates, getting insights on packages, bulk updates, etc. that even if not general enough to be built in in elastic-package, benefit from some of the internal functionality about validation, discovery and overall packages structure knowledge present as part of its internal API.

This tends to end up in having a bunch of adhoc tools reimplementing logic or directly copying it from this repository in order to perform this kind of tasks.

This proposal aims to describe what would be a way to extend elastic-package functionality with custom commands, allowing to reuse all meaningful APIs without the need to bake it in elastic-package itself.

Concept

The proposed concept is to extend the CLI with RPC commands that can be built outside of elastic-package lifecycle. As far as they comply with the described interface and refer to the required elastic-package API version, they will be available in the CLI.

Some of the benefits of an architecture like this are described here.

On the other hand there are some downsides, mainly:

• Defining the plugin types and boilerplate code is a bit tedious, but this is addressed to some extent in the experiment, I would not expect the code to change drastically, maybe only if we decide to do it with gRPC, and even in that case most of it could be reused.
• APIs currently present in internal packages, would need to be exported so they can be referenced by these plugins. This adds maintenance burden down the line, but I think now is a good time to compromise on it before [meta] Path to elastic-package 1.0 #1158 is settled. Doing it after releasing a stable v1 would have much more implications.

Note: The experiment also includes an interactive shell, this could be something desirable or not, but is not necessarily part of the initial scope for this proposal, so please take that into account.

[efd6]

I'm wondering whether it wouldn't be better to use WASM for extensions. These then are safe since the WASM module is then managed by the GC, and don't require RPC. For example with https://github.com/wasmerio/wasmer-go.