Skip to content

Commit

Permalink
fix: Net::HTTP instrumentation no-op on untraced context
Browse files Browse the repository at this point in the history
Update Net::HTTP instrumentation to no-op on HTTP requests within untraced
contexts. Prevents the logging of "called finish on an ended Span" messages.

Example scenario:

The Resque instrumentation can be configured to call
`OpenTelemetry::SDK::TraceProvider#force_flush` at the end of each job. That
causes the batch span processor to immediately send everything it has to the
OTLP exporter. The exporter then marks the HTTP request used to send the data
to the APM server as "untraced", by adding an `untraced` key to the context and
setting it to `true`. Before this commit, the `Net::HTTP` instrumentation did
not check for the `untraced` flag. So it would try set attributes and call
`finish` on a span that was not recording, creating one "called finish on an
ended Span" log entry for each batch of exported spans. On a large application,
that results in millions of those log entries per minute.
  • Loading branch information
brunofacca committed Nov 16, 2023
1 parent 3f44472 commit 0c95add
Show file tree
Hide file tree
Showing 2 changed files with 47 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -86,12 +86,20 @@ def tracer
end

def untraced?
untraced_context? || untraced_host?
end

def untraced_host?
return true if Net::HTTP::Instrumentation.instance.config[:untraced_hosts]&.any? do |host|
host.is_a?(Regexp) ? host.match?(@address) : host == @address
end

false
end

def untraced_context?
OpenTelemetry::Common::Utilities.untraced?
end
end
end
end
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,45 @@
_(span.attributes['net.peer.port']).must_equal(80)
end
end

describe 'untraced context' do
let(:tracer_stub) { Object.new }
let(:expect_in_span_not_to_be_called) do
# Calling `tracer.in_span` within an untraced context causes the logging of "called
# finish on an ended Span" messages. To avoid log noise, the instrumentation must
# no-op (i.e., not call `tracer.in_span`) when the context is untraced.
def tracer_stub.in_span(...)
raise 'tracer.in_span should not have been called'
end
end

it 'no-ops on #request' do
expect_in_span_not_to_be_called

instrumentation.stub :tracer, tracer_stub do
OpenTelemetry::Common::Utilities.untraced do
Net::HTTP.get('example.com', '/body')
end
end

_(exporter.finished_spans.size).must_equal 0
end

it 'no-ops on #connect' do
expect_in_span_not_to_be_called

instrumentation.stub :tracer, tracer_stub do
OpenTelemetry::Common::Utilities.untraced do
uri = URI.parse('http://example.com/body')
http = Net::HTTP.new(uri.host, uri.port)
http.send(:connect)
http.send(:do_finish)
end
end

_(exporter.finished_spans.size).must_equal 0
end
end
end

describe '#connect' do
Expand Down

0 comments on commit 0c95add

Please sign in to comment.