oauth client credentials authentication in http sink

[tareksha]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Sink data to http services protected with oauth2 client credentials flow. Client ID and secret are available as input, and vector should issue bearer tokens and refresh them continuously during runtime.

Attempted Solutions

Two main solution were evaluated:

1. implement a custom http server that acts as an http sink for vector. the server authenticates against the external service and forwards data coming from vector to the external service after adding the bearer header.
   cons: a point of failure, too much complexity for a simple functionality, adds maintenance costs.
2. issue tokens in a separate component that updates vector configuration continuously to modify the "fixed" bearer token in the http sink configuration.
   cons: still a separate component for a seemingly simple functionality, abusing vector's reload functionality.

We prefer to avoid both because they involve writing ad-hoc components that add a point of failure, plus they are too ad-hoc although oauth2 is an industry standard and it fits very will as a built-in feature in vector.

Proposal

Support a new authentication strategy in http sink configuration: "oauth2"

1. auth.strategy now accepts a third value oauth2 in addition to the existing basic and bearer.
2. auth.client_id the oauth2 client ID.
3. auth.client_secret the oauth2 client secret.
4. auth.token_endpoint the endpoint for performing auth2 client credentials authentication.
5. auth.token_refresh_grace a period of time to refresh the token before it expires. for example can default it to 5 minutes.
6. auth.token_client_key
7. auth.token_client_crt

During runtime: Before data is sent to the http sink target a token is obtained from the endpoint. The obtained token is reused for all subsequent request until it expiration time minus the refresh grace is reached, on which a new token is obtained and the process restarts.

The certificate pair in (6) and (7) might be necessary to support token issuers that require a client certificate.

References

No response

Version

No response

[singhbaljit]

This is exactly the feature preventing us from adopting Vector as our edge agent. 👍

Perhaps, we can also do another strategy, openid. The minor change vs. oauth2 is that the HTTP client can discover authentication schemes and endpoints from the standard configuration endpoint.

[tareksha]

Perhaps, we can also do another strategy, openid. The minor change vs. oauth2 is that the HTTP client can discover authentication schemes and endpoints from the standard configuration endpoint.

@singhbaljit client credentials flow is intended to app-to-app communication in oauth-enabled services. can you explain what a general openid helps here?

[singhbaljit]

Right, client credentials flow is still the primary case. The only difference here is what is configured. With OpenID, we just configure the standard OpenID endpoint; the client can discover/validate token endpoint, supported grant types (client_credentials and password), scopes to request.

[tareksha]

Right, client credentials flow is still the primary case. The only difference here is what is configured. With OpenID, we just configure the standard OpenID endpoint; the client can discover/validate token endpoint, supported grant types (client_credentials and password), scopes to request.

i agree with that. but again we are in app-to-app communication and the requested scopes are expected to be set in the configuration. the discovery aspect does not really give a technical advantage.
regarding the token endpoint i can have a rough guess that changing the token endpoint isn't far from changing the discovery endpoint, in practice. i'm finding hard to justify the added complexity of the discovery pre-step.

[singhbaljit]

A few (inter-related) comments:

• I do agree that OAuth2 support with client credentials flow should be the top priority amongst these issues. This will probably unblock majority of people.
• Even though client credentials flow is the recommended flow in machine-to-machine context, I've seen auth servers with such flows disabled many times in my career (i.e. due to security reasons or pricing of such features). In those scenarios, devs end up using password flow. Nothing much changes here from the client's point-of-view, just a few different parameters to pass on the token endpoint. Adding support to define grant type should be eventually added.
• openid support is the lowest on my priority list. OpenID is a popular extension of OAuth2. I think it will would nice to have a native support for it. This allows applies to HTTP sources, especially the HTTP Server source. My only reason to bring this up here was to consider OpenID/OAuth2 more holistically, across all HTTP sources/sinks.

[tareksha]

completed in #21583, many thanks to @KowalczykBartek.

[pront]

Reopening this. Preserving context for future contributors:

Basically someone can re-open #21583 and add a few commits to do this: #21857 (comment).

We (maintainers) will have to follow up and complete the deprecation across multiple releases. I want to thank you everyone for the time spent on this, but we would strongly prefer to make incremental changes since the HTTP sink is a heavily used component.

[lchabert]

As you said, wonderful feature. Do you plan to release it again with changes described in the comment ?

Thx

[pront]

As you said, wonderful feature. Do you plan to release it again with changes described in the comment ?

Thx

This PR #21583 was great and did all the work. It only had a UX problem plus we don't replace existing authentication mechanisms (basic and bearer) in the context of this ticket. We are busy with other work at the moment (including a patch release) but if someone from the community wants to take another crack at this, we will gladly review.