How do we unit test it?

[SuperJMN]

I would like to add tests for my command line parser.
How de we do it? Is there a sample about how to do this properly?

My goal is not to test the whole application, of course (since I'm talking about unit testing), but only that a given command line (string) will fire the expected handler with the correct parameters.

I would also want to emphasize on testing invalid inputs, too.

Thanks in advance and congratulations. This is a great library!

[ashscodes]

Have you seen TestConsole? Here is an example of it being used in the Tests

[JobaDiniz]

How can we test it passing the actual command line, but using stubs or mocks in our services?

public static Task<int> Main(string[] args)
        {
            var parser = new CommandLineBuilder(new RpaCommand())
                .AddInstrumentation() //custom
                .RegisterLoggerFactory() //custom
                .TrackUsage() //custom
                .UseHelp()
                .UseVersionOption()
                .RegisterWithDotnetSuggest()
                .UseTypoCorrections()
                .UseParseErrorReporting()
                .CancelOnProcessTermination()
                .UseExceptionHandler(OnException) //custom
                .AddMiddleware(Middleware) //custom
                .Build();

            return parser.InvokeAsync(args);
        }

I want to test from there, from parser.InvokeAsync("cli command1 subcommand2 --arg1 Hi", new TestConsole()).
Since all the services wiring is done through middlewares:

private static async Task Middleware(InvocationContext context, Func<InvocationContext, Task> next)
        {
            context.BindingContext.AddService<IMyClientFactory>(s => new MyClientFactory(context.Console));

            if (context.ParseResult.CommandResult != context.ParseResult.RootCommandResult &&
                context.ParseResult.CommandResult.Command != null)
                await TryLoadProjectAsync(context);

            await next(context);
        }

I'm trying to create acceptance tests for my cli but without calling external resources like an API (that would be mocked).

[jonsequitur]

Here's one approach that I've used in a few codebases. We expose delegates that will be the implementations for the various commands:

https://github.com/dotnet/interactive/blob/34e0ec750aaca62d1e7900ea2d89490b655f7d52/src/dotnet-interactive/CommandLine/CommandLineParser.cs#L71-L107

Then we replace the defaults with test delegates and capture the parsed and bound parameters for assertions:

https://github.com/dotnet/interactive/blob/a760f11ff2ec94d01881f133cece3e22a1f34ad7/src/dotnet-interactive.Tests/CommandLine/CommandLineParserTests.cs#L55-L76

An example test looks like this:

https://github.com/dotnet/interactive/blob/a760f11ff2ec94d01881f133cece3e22a1f34ad7/src/dotnet-interactive.Tests/CommandLine/CommandLineParserTests.cs#L164-L177

[JobaDiniz]

Thanks for sharing, but it's a bit weird. You created a JupyterCommand but that has nothing to do with the Command Line Command concept. There are delegates, commands, and handlers all over the place.

I hope the next version we will have a better API to deal with testing. Right now, I'm going to borrow your ideas for it, thanks again.

[jonsequitur]

Yeah, that part is pretty weird. The handlers being on e.g. static class JupyterCommand is a vestigial concept that never got renamed and would be better structured as, say, a private local method. The delegates are the interesting part. As long as you expose a way to replace the handler method and the signature matches, you don't need to introduce new abstractions for testability.

The IConsole/TestConsole is the one significant sticky point. If you want to test your app's console output, System.Console is very painful to use, since:

• it's a singleton that will cause parallelized tests to interfere with one another, and
• test runners will redirect output which can also interfere with tests.

[reduckted]

Regarding the IConsole / System.Console problem. I've worked around this by using this approach:

In Program.Main I create a new instance of the Program class that takes an optional IConsole instance to use. If a console is provided, then I use middleware to replace the IConsole instance in the InvocationContext.

public class Program {

    private readonly IConsole? _console;

    public static Task<int> Main(string[] args) {
            return new Program(null).RunAsync(args);
    }

    public Program(IConsole? console) {
        _console = console;
    }

    public Task<int> RunAsync(string[] args) {
        CommandLineBuilder builder;
        RootCommand root;

        root = new RootCommand("My App");
        root.SetHandler(RunCommandAsync);

        builder = new CommandLineBuilder(rootCommand: root);
        builder.UseDefaults();

        if (_console is not null) {
            builder.AddMiddleware(
                (context) => context.Console = _console,
                MiddlewareOrder.Configuration
            );
        }

        return builder.Build().InvokeAsync(args);
    }

}

Then in the tests, I provide a TestConsole instance, which also has the added benefit that I can capture any output and write it to xUnit's ITestOutputHelper.

private async Task<int> TestAsync(params string[] args) {
    TestConsole console;
    int result;

    console = new TestConsole();
    result = await new Program(console).RunAsync(args);

    _outputHelper.WriteLine(console.Out.ToString());

    return result;
}

I think this is a fairly decent approach. The only improvement I can see would be allowing the IConsole to be set in the CommandLineBuilder so that you don't have to use middleware to set it. Something like this would be great:

builder.UseConsole(_console);

or maybe it could be provided in the constructor:

builder = new CommandLineBuilder(console: _console);

[KalleOlaviNiemitalo]

You can pass an IConsole to InvokeAsync.

command-line-api/src/System.CommandLine/Parsing/ParserExtensions.cs

Lines 50 to 54 in 209b724

	public static async Task<int> InvokeAsync(
	this Parser parser,
	string[] args,
	IConsole? console = null) =>
	await parser.Parse(args).InvokeAsync(console);

[reduckted]

🤦‍♂️ Didn't even notice that. Thanks @KalleOlaviNiemitalo !