Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Less verbose/clearer output for pending spec #2956

Closed
eregon opened this issue Apr 23, 2022 · 7 comments · Fixed by #2957
Closed

Less verbose/clearer output for pending spec #2956

eregon opened this issue Apr 23, 2022 · 7 comments · Fixed by #2957

Comments

@eregon
Copy link
Contributor

eregon commented Apr 23, 2022 •
edited
Loading

Subject of the issue

Currently, pending failures are by default shown in a very verbose way, showing the full backtrace and being very difficult to differentiate from real failures (even more so in CI output with no coloring).
Here is an example from https://github.com/rspec/rspec-mocks/runs/6129989849?check_suite_focus=true.
I only post 1 pending (out of 17) otherwise it's near impossible to actually find failures:

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) #any_instance invocation order when setting a message expectation raises an error if 'with' follows 'and_return'
     # see Github issue #42
     Got 2 failures:

     1.1) Failure/Error: expect { expect_any_instance_of(klass).to receive(:foo).and_return(1).with("1") }.to raise_error(NoMethodError)
            expected NoMethodError but nothing was raised
          # /home/runner/work/rspec-mocks/rspec-support/lib/rspec/support.rb:102:in `block in <module:Support>'
          # /home/runner/work/rspec-mocks/rspec-support/lib/rspec/support.rb:111:in `notify_failure'
          # /home/runner/work/rspec-mocks/rspec-expectations/lib/rspec/expectations/fail_with.rb:35:in `fail_with'
          # /home/runner/work/rspec-mocks/rspec-expectations/lib/rspec/expectations/handler.rb:40:in `handle_failure'
          # /home/runner/work/rspec-mocks/rspec-expectations/lib/rspec/expectations/handler.rb:56:in `block in handle_matcher'
          # /home/runner/work/rspec-mocks/rspec-expectations/lib/rspec/expectations/handler.rb:27:in `with_matcher'
          # /home/runner/work/rspec-mocks/rspec-expectations/lib/rspec/expectations/handler.rb:48:in `handle_matcher'
          # /home/runner/work/rspec-mocks/rspec-expectations/lib/rspec/expectations/expectation_target.rb:65:in `to'
          # /home/runner/work/rspec-mocks/rspec-expectations/lib/rspec/expectations/expectation_target.rb:139:in `to'
          # ./spec/rspec/mocks/any_instance_spec.rb:80:in `block (4 levels) in <module:Mocks>'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example.rb:263:in `instance_exec'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example.rb:263:in `block in run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example.rb:511:in `block in with_around_and_singleton_context_hooks'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example.rb:468:in `block in with_around_example_hooks'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/hooks.rb:486:in `block in run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/hooks.rb:624:in `run_around_example_hooks_for'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/hooks.rb:486:in `run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example.rb:468:in `with_around_example_hooks'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example.rb:511:in `with_around_and_singleton_context_hooks'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example.rb:259:in `run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:646:in `block in run_examples'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:642:in `map'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:642:in `run_examples'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:607:in `run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:608:in `block in run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:608:in `map'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:608:in `run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:608:in `block in run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:608:in `map'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:608:in `run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:121:in `block (3 levels) in run_specs'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:121:in `map'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:121:in `block (2 levels) in run_specs'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/configuration.rb:2070:in `with_suite_hooks'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:116:in `block in run_specs'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/reporter.rb:74:in `report'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:115:in `run_specs'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:89:in `run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:71:in `run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:45:in `invoke'
          # /home/runner/work/rspec-mocks/rspec-core/exe/rspec:4:in `<top (required)>'
          # bin/rspec:29:in `load'
          # bin/rspec:29:in `<main>'

     1.2) Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }
            Exactly one instance should have received the following message(s) but didn't: foo
          # /home/runner/work/rspec-mocks/rspec-support/lib/rspec/support.rb:102:in `block in <module:Support>'
          # /home/runner/work/rspec-mocks/rspec-support/lib/rspec/support.rb:111:in `notify_failure'
          # ./lib/rspec/mocks/error_generator.rb:338:in `notify'
          # ./lib/rspec/mocks/error_generator.rb:322:in `__raise'
          # ./lib/rspec/mocks/any_instance/error_generator.rb:7:in `raise_second_instance_received_message_error'
          # ./lib/rspec/mocks/any_instance/recorder.rb:100:in `verify'
          # ./lib/rspec/mocks/space.rb:75:in `block in verify_all'
          # ./lib/rspec/mocks/space.rb:75:in `each_value'
          # ./lib/rspec/mocks/space.rb:75:in `verify_all'
          # ./lib/rspec/mocks.rb:45:in `verify'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/mocking_adapters/rspec.rb:23:in `verify_mocks_for_rspec'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example.rb:525:in `verify_mocks'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example.rb:519:in `run_after_example'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example.rb:283:in `block in run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example.rb:511:in `block in with_around_and_singleton_context_hooks'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example.rb:468:in `block in with_around_example_hooks'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/hooks.rb:486:in `block in run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/hooks.rb:624:in `run_around_example_hooks_for'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/hooks.rb:486:in `run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example.rb:468:in `with_around_example_hooks'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example.rb:511:in `with_around_and_singleton_context_hooks'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example.rb:259:in `run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:646:in `block in run_examples'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:642:in `map'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:642:in `run_examples'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:607:in `run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:608:in `block in run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:608:in `map'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:608:in `run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:608:in `block in run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:608:in `map'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/example_group.rb:608:in `run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:121:in `block (3 levels) in run_specs'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:121:in `map'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:121:in `block (2 levels) in run_specs'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/configuration.rb:2070:in `with_suite_hooks'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:116:in `block in run_specs'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/reporter.rb:74:in `report'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:115:in `run_specs'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:89:in `run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:71:in `run'
          # /home/runner/work/rspec-mocks/rspec-core/lib/rspec/core/runner.rb:45:in `invoke'
          # /home/runner/work/rspec-mocks/rspec-core/exe/rspec:4:in `<top (required)>'
          # bin/rspec:29:in `load'
          # bin/rspec:29:in `<main>' (...)

Failures:

  1) A class double with the doubled class loaded when `.new` is stubbed on a class with a private `new` uses the method signature from `#initialize` for arg verification
     Failure/Error: expect(&block).to fail_with msg
       expected RSpec::Mocks::MockExpectationError with message matching /arguments/ but nothing was raised
     # /home/runner/work/rspec-mocks/rspec-support/lib/rspec/support.rb:102:in `block in <module:Support>'
     # /home/runner/work/rspec-mocks/rspec-support/lib/rspec/support.rb:111:in `notify_failure'
     # /home/runner/work/rspec-mocks/rspec-expectations/lib/rspec/expectations/fail_with.rb:35:in `fail_with'
     # /home/runner/work/rspec-mocks/rspec-expectations/lib/rspec/expectations/handler.rb:40:in `handle_failure'
     # /home/runner/work/rspec-mocks/rspec-expectations/lib/rspec/expectations/handler.rb:56:in `block in handle_matcher'
     # /home/runner/work/rspec-mocks/rspec-expectations/lib/rspec/expectations/handler.rb:27:in `with_matcher'
     # /home/runner/work/rspec-mocks/rspec-expectations/lib/rspec/expectations/handler.rb:48:in `handle_matcher'
     # /home/runner/work/rspec-mocks/rspec-expectations/lib/rspec/expectations/expectation_target.rb:65:in `to'
     # /home/runner/work/rspec-mocks/rspec-expectations/lib/rspec/expectations/expectation_target.rb:139:in `to'
     # ./spec/spec_helper.rb:58:in `prevents'

(I'm investigating the actual failure)

Your environment

  • Ruby version: all
  • rspec-core version: master

Expected behavior

I think pending specs should be shown much more succintly, probably only a single line per pending spec and be trivial to differentiate visually from real failures.
I have been confused multiple times by this and I'm sure others as well.
@pirj
Copy link
Member

pirj commented Apr 23, 2022

I feel your pain.
At least, printing the backtrace for pending issues does not make much sense to me.
When one comes to fixing/implementing the spec, they remove the pending status, and it would print it.
@JonRowe Does it make sense to you? Ok to make such a change in 4.0?

@JonRowe
Copy link
Member

JonRowe commented Apr 24, 2022

A shorter backtrace option for reporting pending specs is certainly something we can do, and could even make it opt-in before releasing 4.0.0

@pirj pirj mentioned this issue Apr 24, 2022
Merged
@eregon
Copy link
Contributor Author

eregon commented Dec 21, 2022

One workaround is to just hide that confusing and verbose Pending: (Failures listed here are expected and do not affect your suite's status) output like #2377 (comment)

@pirj
Copy link
Member

pirj commented Dec 21, 2022

#2957 solves the issue with

RSpec.configure { |c| c.pending_failure_output = :skip }

Would it work for you, @eregon ?

@eregon
Copy link
Contributor Author

eregon commented Dec 21, 2022

Yeah that would be nice and much less hacky if it has the same effect (my understanding is yes).

@pirj pirj closed this as completed in #2957 Jan 7, 2023
@eregon
Copy link
Contributor Author

eregon commented Jan 9, 2023

Do you know if there is a way I get notified when there is a release containing this change?

@JonRowe
Copy link
Member

JonRowe commented Jan 9, 2023

Not sure, I try to go back and mention on issues when doing releases but thats obviously down to remembering to do it 😂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Skip backtrace for pending examples rspec/rspec-core
3 participants
@pirj @JonRowe @eregon