Integrate with Test Anything Protocol or other

[wbazant]

Nice project, I don't have a use case for it but I've read some of it to learn about how you wrote it. As an idea - please only consider if you want to grow the project in such way - would you find it useful to have curl-runnings give out a stream of output, like Test Anything Protocol?

Adding TAP as output would give the suite user reassurance through "I've tested X,Y, and Z" coming up in a log. "I've tested everything, it was okay" is not as easy to work with.

You already have everything it would take:

• runSuite has a list of test cases, and plans to run them sequentially
• each CurlCase can have a simple title, and more details about it - like: operation done - in comments

This opens a way of integrating curl-runnings and its nice DSL with continuous integration systems that give pretty graphs and overview at how often tests are failing. It also lets the test suite be executed by Perl's prove (available on all UNIX based OS's) which has pretty command line output.

An alternative to outputting TAP would be e.g. to mimick JUnit's XML output.

[aviaviavi]

Thanks for this suggestion! I hadn't heard of TAP but this seems like a very reasonable proposal which, as you said, should be pretty simple to do given we already have most of the pieces.

I'm thinking it could just be an additional flag to curl-runnings to have output specifically in TAP compatible format (or any other output format people want), which should only be a small tweak from the output that's there by default.

I myself don't have a need for this integration at this point in time, but if others do I'd happily implement it or obviously accept a PR

[wbazant]

I'm glad you liked the idea! Please do what you want with it. :) Actually I'd make it a core design decision to emit a consistent output stream, and probably also choose one output type, so the tool can focus on the DSL you designed. Maybe one line of structured text format per test, if not TAP. I'd also emit the errors as the queries happen. If the service you're testing is timing out and you've written lots of tests,, you'll know straight away instead of after very long.

[aviaviavi]

If I understand you correctly, we do already output errors as they happen and abort the test suite rather continuing so we should be good there. Or am I missing something that you meant?

A tradeoff with the only one output format is that we currently colorize the output, which I think enhances the readability of the output considerably (especially when using the verbose flag). That alone would break TAP format :/

So yeah I think I'm sold on this being a good feature, but not convinced it should be the only output format or even the default one. Are you aware of a way to make readable / colorized output that still conforms? From doing some a small amount of Googling I'm unsure of how that can be achieved.

[wbazant]

I'd make the output as predictable as can be in the spirit of a design philosophy (that you don't have to share) to make the program do one thing well, and let it collaborate with other tools:

• Say something for the passed tests, and let another tool select the failure cases

• Make the standard output contain everything that's useful to the user - comments about what request got done, and asserts that got made (if you can undertake to make them all implement show, which might be a tall task).

• Make all the other logs go to STDERR - the ones for developers who need to debug the program. The --verbose flag is really for them.

• Don't terminate early on errors, your program will very likely terminate and stop sending requests after receiving SIGPIPE, and then you can add "early termination" when you pipe the output somewhere else:

for i in pass pass fail pass; do echo $i; echo "Process 1 said:" $i >&2 ; sleep 1 ; done | perl -e 'while(<>) { print "Process 2 saw: " , $_; /fail/ and exit; }'
Process 1 said: pass
Process 2 saw: pass
Process 2 saw: pass
Process 1 said: pass
Process 2 saw: fail
Process 1 said: fail

Re colours- you could detect whether the output pipe is a TTY. I think your library already does this. Compare with how grep behaves:

grep --color lo <<< 'Hello!' # with color on my terminal
grep --color lo <<< 'Hello!' | cat #loses color

[aviaviavi]

This sounds reasonable to me. The stderr thing is a good idea for this, I think that's a good way to have extra output that doesn't mess with the standardized formatting.

Cool well there's some other features in the queue that need to get done, but I consider this something that will be added when time permits!

And to anyone else reading through this issue, feel free to reach out if you feel up for implementing this before I get to it.