Hide traceback and verbose otter output to only show assertion message instead

[joelostblom]

Is it possible to hide the traceback of the tests and just show the message from the assertion of the test that failed? Showing all this output to students will likely be quite confusing and it also allows them to see exactly what is tested which to me is less desired than just showing the assertion message with a hint of what is wrong. This is a bigger problem when there are multiple assertions in the same cell.

I have seen the fail_message option in the question metadata, but it does not suppress the traceback and requires manually inputting the same string that is already in my assertion message so it is not ideal.

[joelostblom]

Since the test namespace is isolated from the rest of the notebook I tested with import sys; sys.tracebacklimit = 0 which slightly cuts down the printed messages.

Before:

After:

However, I would like to suppress the rest of the printed output as well, and I updated the title of this issue to reflect that.

[chrispyles]

This wouldn't really work in all cases. Otter's test cases rely on doctests, so not all failed test cases raise errors. For example,

>>>  my_ans == 1
True

Without a substantial rewrite of the doctest running infrastructure in Otter, I don't think this is possible, especially when you consider adding a way to make this behavior configurable since many instructors will want to keep the existing behavior.

[joelostblom]

Maybe it would be possible via the different verbose flags for doctest? There are a few mentioned in the documentation and I will try to look into it myself also. A possibly simpler solution might also be to suppress all doctest output and just show the failure message, this would serve the purpose of giving students a hint without giving away the answer.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[joelostblom]

@chrispyles I discovered that it is actually quite straightforward to introduce this behavior. By not returning runresults.getvalue() when the function run_doctests() fail we can avoid printing all the verbose info from doctest. By adding a fail message, the output could become something like this:

Adding emojis makes this output even easier to scan through and quickly identify the failing tests:

I think this really makes it easier for students to find the helpful fail message and it does not give away the exact test used as well as looks less intimidating than the current default behavior which prints a lot more text:

Would you consider reopening this issue and reviewing a PR that introduces an option to enable this behavior?

(ref #429 for myself when trying to implement this)

[chrispyles]

@joelostblom happy to.