Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Existing sampling decision ignored when creating new segments with no http.Request (SQS -> ECS tracing) #350

Open
okonos opened this issue Mar 23, 2022 · 1 comment

Comments

@okonos
Copy link
Contributor

okonos commented Mar 23, 2022

To implement tracing on an ECS worker processing SQS messages, we use the NewSegmentFromHeader function:

func NewSegmentFromHeader(ctx context.Context, name string, r *http.Request, h *header.Header) (context.Context, *Segment) {

Since there's no http.Request available in a SQS processing context, a nil is passed instead. This causes the existing sampling decision to be ignored and a new one requested instead:

if r == nil || traceHeader == nil {
// No header or request information provided so we can only evaluate sampling based on the serviceName
sd := seg.ParentSegment.GetConfiguration().SamplingStrategy.ShouldTrace(&sampling.Request{ServiceName: name})

This results in some segments missing and possibly in depleting the sampling reservoir, which could be used by other traces.

Looking through the docs and code there doesn't seem to be any alternative way of creating a segment from an existing tracing header. A workaround we've come up with is to pass an empty request (&http.Request{URL: &url.URL{}}) to the NewSegmentFromHeader function.
This could preferably be solved on the SDK level, though. A relatively simple solution would seem to be to remove the r == nil condition from:

if r == nil || traceHeader == nil {

and instead fill the sampling.Request with HTTP request related data in both if branches only when it's available. After all, the HTTP request is only used when requesting the sampling decision.

Any insight into the issue and proposed fix would be much appreciated.

Possibly related issue: #234

@bhautikpip
Copy link
Contributor

Yes, I agree with the temporary fix you suggested and suggestion you provided for permanent fix as well. I will mark this as an enhancement. Yeah, we can probably remove the r == nil check to include use cases like yours.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants