Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DRAFT] Skeleton changes for new memory limiter models #12558

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

jmacd
Copy link
Contributor

@jmacd jmacd commented Mar 5, 2025

Description

Introduces the changes described in the comment open-telemetry/opentelemetry-go#6404 (comment). This adds configuration to the memorylimterextension package with a model ("gcstats" or "admission") and a new sub-struct containing the admission-control-based configuration.

This also adds a Controller interface with a no-op implementation. The real implementation of this is in collector-contrib/internal/otelarrow.

Our goal is to automate the insertion of gRPC and HTTP-specific interceptors to make it easy to ues the controller interface defined here. I believe the way we will do this is to modify the existing code points, such as inside (*configgrpc.ServerConfig) getGrpcServerOptions(). In these locations, we will apply an inteceptor to count the bytes of the request and call the controller to:

   requestSize := computeRequestSize(req) // This could have several cases
   releaseFunc, err := controller.Acquire(ctx, requestSize)
   if err != nil {
      return err
    }
    defer releaseFunc()
    // call the next handler here

I'm not sure precisely how the controller will be injected to the handler used in the receivers. In other words, while constructing the receivers we want to easily (for all components) inspect the Host for a memorylimiter extension, which will be either GCStats or Admission-based, but either way implement the controller interface. The interceptors will be provided with the controller, etc.

Note that some receivers will require direct access to the controller because they do not use interceptors, or because they use custom gRPC protocols with internal compression. For example, a Prometheus receiver would call controller.Acquire() directly. For example, the OTel-Arrow stream receiver decompresses each stream request and Acquires the uncompressed size before returning from Send().
  

Link to tracking issue

Part of #9591.

Testing

The OTel-Arrow components, where the Admission-based logic originates, have an end-to-end test of the behavior of this component.

Documentation

TODO: The README.md for memorylimiter extension refers to the memorylimiter processor. This situation should be reversed. The memorylimiter processor shall be scheduled for removal after these changes are complete.

@jmacd jmacd requested a review from a team as a code owner March 5, 2025 01:34
@jmacd jmacd requested a review from jade-guiton-dd March 5, 2025 01:34
@jmacd jmacd marked this pull request as draft March 5, 2025 01:35
@mx-psi mx-psi self-requested a review March 5, 2025 11:22
@mx-psi
Copy link
Member

mx-psi commented Mar 5, 2025

Self-requesting review, will review once you mark it as ready/say so in a comment :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants