Further enhancing is_deeply diagnostics

[philiprbrenan]

In my frequent daily use of Test::More I find is_deeply to be highly relevant, because when things go wrong, it prints out helpful diagnostics. So I use it to compare strings and numbers as well as structures to speed up my diagnosis of failing tests. Indeed I prefer it to the ok series because it prints these diagnostics while ok does not. When comparing strings in this manner it would be helpful if is_deeply could indicate the location of the first mismatch in the printed strings received and expected as it often takes quite a lot of effort to locate this point manually in long strings. In my tests this is often at the end of the string where perhaps the test expects a trailing end of line but the actual result received is missing said new line, or has two too many. Might it be possible to further enhance this excellent function to further speed my test driven development technique which rely so heavily upon is_deeply's capabilities?

[exodist]

This is what Test2 is for:

• The Test2::V0 bundle includes an is() that replaces is_deeply() and provides much better diagnostics
• You could also load Test::Tools::Compare to load just the new is() without the rest
• If we need to make a change that breaks backwards compatibility we can release a Test2::V1 and not change Test2::V0

Test2 was written specifically because Test::More had serious limitations on how it can be changed. When I first inherited Test::More from Schwern I was of the opinion that we should not be afraid to break things, and should improve the tools in Test::More directly to be better. The first thing I tried was fixing diagnostics messages thinking "What is the worse that could happen?" The answer: An ungodly amount of downstream modules implement custom test tools that wrap is, ok, is_deeply, etc, through in a local $Test::Builder::Level = $Test::Builder::Level + 1 and then test the new test tool by making giant string comparisons of both success and failure cases. So the result is that even tiny changes to diagnostics produced by the tools in Test::More will break several critical cpan modules, and even more less critical ones.

This realization, that any changes to Test::More would require cooperation of many authors, some of whom are absent, and would break everything otherwise, is what resulted in Test2, and more importantly Test2::Suite and Test2::V0 existing.

• The Test2 API was written specifically so that people can validate their own testing tools without using string comparison on TAP and comment output.
• The Test2::VX bundle system was written to make a clear path foreword for when we do need to make breaking changes we can do so while also not breaking existing things.
• Test::More now does actually use Test2 under the hood, to be more correct, Test::Builder has been a compatibility layer on top of Test2 for several years now. As a result almost all Test::Builder based tools play nicely with Test2, so you can mix and match. This was an incredible amount of work as I was not willing to break anything important in the process, and managed in the end to only have a handful (~3 I think?) of unimportant cpan modules break from the change.

Test::More is now considered "on life support". I will not make breaking changes unless there is no option. Breaking changes include diagnostics changes sadly. Critical bugs, or fixing things that will break because of larger perl changes, etc still happen, but that is about it.

We could argue all day about how we should not have gotten here, that people making big string comparisons on TAP is a mistake and they can reap what they sow, I will probably agree with everything someone might say about this... but at the end of the day we are here, we were here when I inherited the project, and I am not going to break cpan for everyone.

If you do not want to switch to Test2, that is fine, I am not going to apply any pressure there, so do what works best for you. There are also other cpan libraries I had nothing to do with that implement alternative is_deeply functionalities if anyone decides to boycott Test2 for whatever reason.

Sorry in advance if you do not like this answer.

[exodist]

Sorry if that response is saltier than the question warrents. I have had this discussion a lot lately with people who either do not know about Test2, or do not want to bother looking at it. So I unloaded all the reasoning and results of previous discussions on the matter in hopes that if the discussion here does continue it can hopefully be original content, and not the same discussion I have had countless times.