Could you share the repo for me to debug?

Unfortunately it’s not a public repo.

That's ok. I have reproduced the issue on my end. After some initial exploration, it appears that this is "standard" Rails behavior; that is, in production Rails doesn't report exceptions to the error reporter when the errors occur within the HTTP request lifecycle. The short answer, for now, is that in production Rails renders exceptions instead of raising exceptions, and the current code only reports raised exceptions.

I am trying to find someone who might know if there is any way to configure Rails to report rendered exceptions in production. There might not be such a config presently tho. If not, then I will work to open a PR to Rails to fix this major limitation of the error reporter. But, for now, there is no fix I can find. I will keep you updated here as I learn more tho.

@holman: Can you test that this branch solves your problem? Update your Gemfile with this:

gem "solid_errors", github: "fractaledmind/solid_errors", branch: "patch-executor"

I don't like that I have to patch a Rails internal, but for now, that is the solution I can find.

Pretty wild hole we're in here!

I switched the branch, but strangely not seeing any difference.

Now that is interesting, because it is working for me in production. If you get the chance to debug, you could add some manual logging to the patched Executor and see [1] if the patched Executor is being called, [2] if there is an action_dispatch.exception present, [3] if yes to both, if the error reporter present at that line is the one with the SolidErrors subscriber.

In my case, the problem simply that Rails only reports raised exceptions, and it swallows exceptions in production to render error pages. This patch to the executor to pull out any logged exceptions (stored in the action_dispatch.exception header) and report them is correctly storing errors and occurrences for me in my production application. So, I'm not immediately sure what might be different for you. I will need your help to debug on your side.

I just connected this conversation to the issue that I have opened in Rails. Hopefully, someone at some point will get involved there and we can start making our way to a solution in Rails itself, because to my mind this is clearly a Rails bug. An error reporter that doesn't report errors in production is just about useless. I never expected it to not work in production (and, I didn't realize that it wasn't working because I was seeing errors from background jobs and just figured there weren't any production errors occurring 😅. So thank you for bringing this to my attention)

Hah, yeah no worries! It was pretty surprising to me, too. Will be interesting what comes out of the Rails issue; I’m subscribed there too.

Unfortunately I’m heading out for a few hours, but will dig into it more on my side later today. Thanks for all the digging thus far!

Have you looked into how Sentry works? I thought I remembered them working on changes to support Rails' new error subscriber stuff, maybe here: https://github.com/getsentry/sentry-ruby/blob/master/sentry-rails/lib/sentry/rails/railtie.rb#L51 via getsentry/sentry-ruby#1705 -- I haven't spent much time looking into this, but I noticed they have app.executor.error_reporter.subscribe for example.

I dug into Honeybadger's source, but not Sentry's yet. I expect it to be similar. Honeybadger adds a middleware which pulls out the exception from the header.

Looks like the problem (at least in my setup) is this line:

ActiveSupport.on_load(:action_dispatch) do

That never seems to get loaded looks like? So it doesn't end up calling the ActionDispatch::Executor block. The monkeypatch does get loaded otherwise, though. (The higher-up initializer "solid_errors.active_support.notification_subscriber" block does run.)

Right, just double checked and that is not a load hook that Rails actually calls. Just updated the code in that branch. Also going to keep digging and experimenting to see if we can make this not necessary.

Fixed in edge Rails!

Is rails ~> 7.0.1 also impacted by that issue? I have a similar case in which only exceptions raised inside Rails.errors.record are being saved by solid_errors