Capturing x-amzn-RequestId for X-Ray Tracing

[sarmst22]

I'm building a rust lambda and I'm attempting to leverage the X-Ray functionality to trace a downstream dynamo call. I've built a Segment Document, and successfully submitted it and a subsegment for a dynamo call to X-Ray.

However I can't seem to find where I would retrieve the x-amzn-RequestId from the dynamo client response to add to the X-ray Subsegment.

The value would be passed in as a request_id in the current json version based on the example here (https://docs.aws.amazon.com/xray/latest/devguide/xray-api-segmentdocuments.html#api-segmentdocuments-aws)

{
  "id": "24756640c0d0978a",
  "start_time": 1.480305974194E9,
  "end_time": 1.4803059742E9,
  "name": "DynamoDB",
  "namespace": "aws",
  "http": {
    "response": {
      "content_length": 60,
      "status": 200
    }
  },
  "aws": {
    "table_name": "scorekeep-user",
    "operation": "UpdateItem",
    "request_id": "UBQNSO5AEM8T4FDA4RQDEB94OVTDRVV4K4HIRGVJF66Q9ASUAAJG",
  }
}

request_id: "UBQNSO5AEM8T4FDA4RQDEB94OVTDRVV4K4HIRGVJF66Q9ASUAAJG"

Is there a current way to gather this data in how responses are deserialized? Or do I not need this for proper tracing?

[rcoh]

There isn't a first-class way to get that right now unfortunately. You'll need to use a workaround by going through the low level DynamoDB client. There, you can use .call_raw() instead of call. This will give you access to the raw HTTP response. The field you need will be in the X-Amzn-Requestid header.

    let conf = aws_sdk_dynamodb::Config::builder()
        .region(Region::new("us-east-1"))
        .build();
    let raw_client = aws_hyper::Client::https();
    let update_item = aws_sdk_dynamodb::operation::UpdateItem::builder().<...>.build().unwrap().make_operation(&conf).unwrap();
    let resp = raw_client.call_raw(update_item)?;
    let request_id = resp.raw.headers.get("X-Amzn-Requestid");
    let parsed_response = resp.parsed;

We're looking into exposing the request id in success and error responses to make this nicer in the future.

[sarmst22]

I'll give that a shot, exposing the request id would be a massive boon for this in the future improving the accessibility of X-Ray, I look forward to it getting easier to access.

[sarmst22]

@rcoh content_length and the status would also be helpful in this specific case, are they also being examined as candidates to be easier to access?

[rcoh]

What is your use case for those specific values? If you're operating on the raw response info like status, it's possible X-Ray integration would be better handled as a middleware stage (something else we can investigate)

…

On Mon, Aug 16, 2021 at 3:14 PM sarmst22 ***@***.***> wrote: @rcoh <https://github.com/rcoh> content_length and the status would also be helpful in this specific case, are they also being examined as candidates to be easier to access? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#194 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADYKZY7J26CIULVMU6I3RLT5FPRNANCNFSM5CIJGPJQ> .

[mbergkvist]

If you're operating on the raw response info like status, it's possible X-Ray integration would be better handled as a middleware stage (something else we can investigate)

@rcoh What is the status on this? A middleware for X-Ray integration for the SDK clients would be a really nice feature to the SDK.

[rcoh]

A middleware for X-Ray integration for the SDK clients would be a really nice feature to the SDK.
related discussion: #504

I totally agree—it's not on our roadmap right now unfortunately but:

1. It's possible to add your own middleware to the SDK fairly easily by using map_middleware on aws_smithy_client::Client:

    let inner_client = aws_smithy_client::Builder::new()
        .https()
        .middleware(AwsMiddleware::default())
        .map_middleware(|middleware| {
            ServiceBuilder::new()
                .layer(MapRequestLayer::for_mapper(MyMiddleware))
                .layer(middleware)
        })
        .build();
    let client = aws_sdk_sts::Client::with_config(inner_client, sts_config);

2. We would also happily accept a PR to add it to the default middleware stack: https://github.com/awslabs/smithy-rs/blob/9708aa830730956a5a5a1500c37a6f95fffc9bfb/aws/rust-runtime/aws-inlineable/src/middleware.rs#L56

Note that the RecursionDetection middleware does actually forward the x-amzn-traceid but it seems like there is still at least one missing piece to get x-ray totally working. #504

[sarmst22]

Updates on my end of things regarding this @rcoh; I've switched to leveraging the AWS ADO Collector distro and the Opentelemtry for Rust implementation with the tracing.rs crate's integration with Opentelemtry (OTEL).

This has allowed us to capture a Rust Lambda's x-amzn-traceid values and translate them into parent contexts for our OTEL Traces, which then correctly exports function instrumentation data to AWS X-Ray. While we still have the problem of connecting Lambda Invocations trace id values together when initialized via SQS connections, and inferred X-Ray segments need to be manually created (something I'm actively working on), it does allow us to capture function information from Lambdas in X-Ray.

[sarmst22]

@rcoh When we're talking about this Middleware, would we just be creating something to attach the current X-Ray Trace Id into the client Request headers?

Or are you envisioning a Middleware that leverages the current X-Ray functionality of the SDK to directly post X-Ray Segment Document data to the X-Ray api alongside trace id propagation?

[rcoh]

I actually haven't dug into it deeply—we need to do some more work on the observability story of the Rust SDK. Posting to X-Ray directly is probably not something we'd want to do by default, but we could maybe enable it on an opt-in basis?

If you have thoughts on what would make sense from the customer perspective on this, I'd encourage you to write an RFC if you were interested.

https://github.com/awslabs/smithy-rs/blob/main/design/src/rfcs/overview.md

[Velfi]

Getting the request ID from an output or error was implemented in this PR.

For outputs:

// Import the trait to get the `request_id` method on outputs
use aws_sdk_s3::types::RequestId;
let output = client.list_buckets().send().await?;
println!("Request ID: {:?}", output.request_id());

For errors:

use aws_sdk_s3::types::RequestId;
println!("Request ID: {:?}", error.request_id());

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.

[inickles]

Just noting for future reference that RequestId has apparently been moved from the types module to the operation module:

// Import the trait to get the `request_id` method on outputs
use aws_sdk_s3::operation::RequestId;
let output = client.list_buckets().send().await?;
println!("Request ID: {:?}", output.request_id());