Run same tests with multiple scenarios

[AlexanderLanin]

I have a bunch of tests in test/my-feature/test.sh which I want to test with a bunch of base images (with python, without python, etc).

I could run this line with a bunch of different base images:
devcontainer features test --features my-feature --base-image mcr.microsoft.com/devcontainers/base:ubuntu .

But it seems this is exactly what scenarios are for:

{
    "base_ubuntu": {
        "image": "mcr.microsoft.com/devcontainers/base:ubuntu",
        "features": {
            "my-feature": {}
        }
    }
}

However from base_ubuntu.sh I cannot source test.sh or any other script I have since it's fully isolated.
Obviously I don't want to copy paste the contents of test.sh to a bunch of different files.

So how do I run the same script from different scenarios?

[Chuxel]

@joshspicer At one point referencing an image didn't work. Did this get fixed with your most recent updates? I ran into this as well (along with being able to specify the remote user)

@AlexanderLanin I think right now you end up having to copy the same script name multiple times. If you are on linux or macOS, you could use a symlink.

[joshspicer]

The two current ways to execute tests with that command are:

1. The "autogenerated test", which use the --features <ARRAY_OF_FEATURES> and --base-image <IMG> to dynamically creates a test .devcontainer.json with all the ARRAY_OF_FEATURES (with default parameters) built on top of `IMG.

2. "Scenarios", which lets you encode an entire devcontainer.json (including image, etc) instead of using the --base-image flag.

---

Perhaps the devcontainer feature test command should copy the entire /test directory into the temporary directory?

Take the python Feature's test dir:

Perhaps when running a scenario (say install_jupyterlab.sh), the entire contents of this folder should be copied over to execute the test. By default the test runner will execute ./install_jypyterlab.sh for this scenario, but then this script could source any of the other scripts, etc.

@Chuxel @AlexanderLanin would this be a useful improvement?

[Chuxel]

Yeah, that would be good since you can add test data files at that point as well. e.g. for the nix feature I use a remote Flake rather than a local one because of that.

[AlexanderLanin]

For the moment / for the presented problem, that would help me as well!

However now I have some concerns how this will scale to some dozen scenarios: my current (extremely limited!) experience shows that [scenario].sh files are mostly identical for every scenario. Copying the entire test directory certainly allows me to source my common_test.sh from every [scenario].sh file, but I might end up with lots and lots of single line .sh files.
Maybe that's a separate discussion... maybe not.

Currently I still believe in the idea of having a "test_script" entry within scenario specification (+copying the entire test directory).

[joshspicer]

Good feedback - I agree. I'm going to make both changes (already started on the first one, and will look to add something like a _test_script property to the individual scenarios).

[joshspicer]

This is the change and a validating test for the first solution. Going to batch this with more changes in devcontainers/cli#265