Feature req: Remove pending examples details by default

[marko-avlijas]

Reasoning

I like to work using guard and running file I am working on in documentation mode, then if it passes running all tests.

Sometimes I will write a serious of specs using just it with no block so I can think about the subject. My concentration is needlessly broken with verbose pending output and yellow colors, so I tend to comment out pending examples and uncomment them as I implement them which is waste of time.

Solution

Default output already tells me I have pending examples. Just remove the details.

Finished in 0.66204 seconds (files took 0.77603 seconds to load)
738 examples, 0 failures, 10 pending 

If someone wants to see all pending examples in his source code, he can run last command with option --show-pending

Alternative

If you want to keep current output by default then perhaps if we can get an option like this.

rspec --no-pending-details

[myronmarston]

AFAIK, you're the first to request that we hide the pending example output. I don't think it makes sense for us to bloat the CLI with an option for this. If we implemented every feature like this requested by a user, our CLI would be very large and bloated, and our maintenance burden would be very large.

You can implement this for yourself pretty easily:

module FormatterOverrides
  def example_pending(_)
  end

  def dump_pending(_)
  end
end

RSpec::Core::Formatters::DocumentationFormatter.prepend FormatterOverrides

Or if you only want to silence block-less examples:

module FormatterOverrides
  def example_pending(notification)
    super if notification.example.metadata[:block]
  end

  def dump_pending(_)
  end
end

RSpec::Core::Formatters::DocumentationFormatter.prepend FormatterOverrides

Or, if you just want to filter out block-less pending examples (but still show other pending examples):

RSpec.configure do |c|
  c.filter_run_excluding :block => nil
end

The point is, RSpec is very flexible, so users who want features like this can implement it themselves. We designed the formatter API so that anyone who wants custom format can implement a formatter (or prepend a module to change an existing one, as I did above).

[andy-twosticks]

I confess I am surprised that this was closed so readily. I and the OP cannot surely be the only people who define tests before we actually write the code inside them? And anyone that does will get a long list of pending warnings with some test output buried somewhere inside it?

At the very least the solution above should be documented somewhere? I have my own version, which is far more fragile, and it took me a while to create...

[marko-avlijas]

Thank you for the fix.

You are right, adding bloat to command line options is not desirable.
Perhaps add option to RSpec configuration in spec_helper?

I understand if that's too much trouble for little demand.

[myronmarston]

I and the OP cannot surely be the only people who define tests before we actually write the code inside them?

Nope, I do this all the time, and I've always been happy with the output for this situation. I specifically like seeing the descriptions of each example (even ones I haven't filled in the block for yet) so I an make sure they are worded well when combined with the group description(s). Maybe lots of people are unhappy with it, but I had never heard anyone mention this issue until yesterday.

I provided a couple of solutions, including a dead-simple one-liner:

RSpec.configure do |c|
  c.filter_run_excluding :block => nil
end

Is there any reason that solution I provided is insufficient?

Closing issues isn't necessarily forever (we often re-open them) but if we don't close them when we think the user's issue is addressed they tend to sit open semi-permanently and then @xaviershay winds up spending an afternoon pruning our issues like he did recently :).

@marko-avlijas, I don't think a config option makes sense for RSpec because how results are displayed in the output is a property of the formatter, not a global property of RSpec. RSpec is designed to support extremely flexible formatter output. If you consider just the main two built in formatters (progress and documentation) the yellow dot in the progress formatter is probably not something you'd want changed, but the documentation output you do want changed. There are tons of 3rd party formatters and it's not clear which you'd want this to apply to (nor would it really be feasible to apply this to them), so we'd wind up with a config option that feels global (because it's available off of RSpec.configure) but for most formatters does nothing.

If you don't like the filtering solutions, you can write your own formatter; that's why the formatter API exists. Since we can't please everyone with our formatter output we made it self-serve so you can define your own formatter to get any output you like.

Does that work for you, @andy-twostics and @marko-avlijas ?

[xaviershay]

Our policy is to not leave issues open unless there are clear next steps. Re-opening is fine! As myron mentioned, we have a very high bar for new features to RSpec, instead preferring features to "prove themselves" as plugins. It's how we keep any kind of sanity maintaining it ;)

You mentioned wishing for better documentation: we're always open to improving it :) Where were you looking that you couldn't find it?

[marko-avlijas]

Your fix does work. I have also added the same thing for progress formatter.

RSpec::Core::Formatters::ProgressFormatter.prepend FormatterOverrides

You are doing a great job with rspec. Just go on.

I didn't know you can put _ as param list. Learned something new. :)
Does that do something special or is it just convention when you don't care about params?

[JonRowe]

_ is the convention for a param you're not going to use, formatter methods only take one param, which is a notification object.

[matkoniecz]

@myronmarston Thanks for code but I have problems with using it.

At start I tried puting it in the most obvious place - spec_helper.rb

 module FormatterOverrides
  def example_pending(_)
  end
  def dump_pending(_)
  end
end

RSpec.configure do |config|
  RSpec::Core::Formatters::DocumentationFormatter.prepend FormatterOverrides  
(...)
end

But it failed to change anything. Now I found https://www.relishapp.com/rspec/rspec-core/docs/formatters/custom-formatters and created

module FormatterOverrides
  def example_pending(_)
  end

  def dump_pending(_)
  end
end

class CustomFormatter
  RSpec::Core::Formatters::DocumentationFormatter.prepend FormatterOverrides  
end 

file and used rspec spec/recipe_splitter_spec.rb --require ./spec/custom_formatter.rb --format CustomFormatter but I was hit with

 The CustomFormatter formatter uses the deprecated formatter interface not supported directly by RSpec 3.  To continue to use this formatter you must install the `rspec-legacy_formatters` gem, which provides support for legacy formatters or upgrade the formatter to a compatible version.  Formatter added at: /home/mateusz/.rvm/gems/ruby-2.3.3/gems/rspec-core-3.5.4/exe/rspec:4:in `<top (required)>'

Where I can find documentation for how one may use custom formatters?

[JonRowe]

https://relishapp.com/rspec/rspec-core/v/3-5/docs/formatters/custom-formatters

[marko-avlijas]

@matkoniecz move that prepend statment out of configure block.

This is my spec_helper.rb

RSpec.configure do |config|
   ...
end

# Silence output from pending examples in documentation formatter
module FormatterOverrides
  def example_pending(_)
  end

  def dump_pending(_)
  end
end

RSpec::Core::Formatters::DocumentationFormatter.prepend FormatterOverrides

# Silence output from pending examples in progress formatter
RSpec::Core::Formatters::ProgressFormatter.prepend FormatterOverrides

[JoshCheek]

This is what I wound up with:

# Modifying output b/c the skipped tests are spammy since we skipped tons of entire
# suites while switching over to environments. This implementation is based on
# https://github.com/rspec/rspec-core/issues/2377#issuecomment-275131981
module RSpec::Core::Formatters
  module MyAppOverrides
    # Don't display the summary at the end
    def dump_pending(*)
    end

    # Don't display the skip message. Overrides this method in the DocumentationFormatter:
    # https://github.com/rspec/rspec-core/blob/v3.10.1/lib/rspec/core/formatters/documentation_formatter.rb#L79-L83
    private def pending_output(example, _spammy_message)
      ConsoleCodes.wrap("#{current_indentation}#{example.description.strip}", :pending)
    end
  end

  constants.map { const_get _1 }.each { _1.prepend MyAppOverrides if _1 < BaseTextFormatter }
end

[dogweather]

This is what I wound up with:

What file does that go into? Is there other config necessary to enable it?

[pirj]

@dogweather Any file that loads along with tests. Easiest is to add this code to spec/spec_helper.rb.
Let's continue the discussion in our mailing list https://rspec.info/help/

[eregon]

FWIW here is what I did in concurrent-ruby to hide the pending output at the end: ruby-concurrency/concurrent-ruby@0cd4085

# Remove verbose and confusing "Pending: (Failures listed here ...)" section at the end.
# From https://github.com/rspec/rspec-core/issues/2377#issuecomment-275131981
module RSpecNoPendingOutput
  def dump_pending(_)
  end
end
RSpec::Core::Formatters::DocumentationFormatter.prepend RSpecNoPendingOutput

That output is so confusing in CI logs, it looks like errors when it isn't.

[dogweather]

This has me wondering about implementing the open/closed principle architecturally for tool configuration & output modification. This is the second tool I use on a daily basis with a closed-won't-accept-PRs issue like this, and closed for the same reason: the tool works fine for the author, and the author doesn't see enough support behind the change to merit accepting a PR for the feature.

I'm not arguing whether or not the decision to reject is valid: instead, I'm brainstorming about coding strategies to maintainably enable these enhancements without changing the core code.

[pirj]

The PR that addresses "remove pending examples output" is #2957, we're considering merging it.
And it's not a specific case, see e.g. #2978 that we're happy to accept a PR for making the output more flexible.

[JonRowe]

Instead, I'm brainstorming about coding strategies to maintainably enable these enhancements without changing the core code.

Output in RSpec is controlled by formatters with a defined API that passes notification objects containing a lot of the commonly used building blocks for output, if you have custom output requirements its easy to implement your own formatter, you can even base it off ours (although we prefer you vendor a copy and tweak it as required rather than inheriting purely to guard yourself against implementation changes).