Agent-api changes for Dekaf

[jshearer]

Description:

This was originally part of #1791 but I pulled it out in order to get it merged sooner, as the changes to support Dekaf materializations require these changes.

• Introduce /authorize/dekaf, an endpoint designed to give Dekaf all of the pieces it needs to serve requests. These are:

  • A short-lived control-plane access token to authorize requests under the dekaf role which has grants to the public.registered_avro_schemas table. This works by taking advantage of Postgrest's JWT auth, where a properly signed JWT with a role claim will cause all API requests using that token to execute under the corresponding Postgres role.

    Note: This is how our current user/RLS-based auth works, except that all tokens use the authenticated role and then are further scoped by transaction-scoped settings

    The dekaf role is defined as:

     CREATE ROLE dekaf NOLOGIN BYPASSRLS;
     GRANT dekaf TO authenticator;
     GRANT USAGE ON SCHEMA public TO dekaf;
     GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO dekaf;
     GRANT SELECT,INSERT,UPDATE,DELETE ON public.registered_avro_schemas TO dekaf;

  • The models::MaterializationDef for the materialization being requested as identified by the sub JWT claim. This will both allow Dekaf to validate that the request being made has the proper bearer token, and will provide information about which topics (bindings) to expose.

  • The ops logs and stats journal names for the materialization. This will allow Dekaf to write ops logs and stats for operations as well as billing.

• Updates /authorize/task to support requesting a token containing both READ and LIST capabilities
• Updates flow-client to :
  • Rename the existing fetch_task_authorization() (which actually calls /authorize/user/task) to fetch_user_task_authorization()
  • Add a new fetch_task_authorization() which calls /authorize/task

---

This change is 

[jgraettinger]

Ah I see, role is restricted through deserialization.

This feels a bit weird to me, because I cannot conceive of a likely scenario where we would want to extend this, and thus don't need this generalization.

Stepping back for a moment to look at this more holistically: the runtime gets a bunch of metadata through Etcd-stored ShardSpecs, like the build ID to use which unambiguously resolves / allows for loading the task spec, and also the name of the ops logs & stats partition to use, which are encoded as shard spec labels, plus some other stuff.

Dekaf tasks don't have shards, we need an alternative for this discovery, so a new API is warranted. Somehow we need to:

• Give Dekaf a JWT which allows it to work with the AVRO schemas table
• Let Dekaf know the exact materialization spec to use.
• Tell Dekaf what the ops logs / stats partitions are to use.
• Probably other stuff we may not have thought of.

I think my central feedback is that the /authorize/role abstraction is painting us in a corner, and instead we want an /authorize/dekaf which embraces the differences between how Dekaf bootstraps (without shards) and how the runtime does.

It could directly load / return the current MaterializationSpec, for example, which removes the RLS question for live_specs from the dekaf role JWT and allows us to narrow it's access. It could return the build ID and other metadata for associated logging within Dekaf. It can tell dekaf of the ops logs / stats partitions to use, which only needs to be resolved once and not for every task binding (which is part of why it's odd for it to be included in /authorize/task). It would be less gross to extend if we discover other bits that Dekaf can benefit from.

[jshearer]

Yeah, I went back and forth on this and ended up going with the more general /authorize/role because /authorize/dekaf felt like a deviation from the level of generality that everything surrounding it exists in. That is, the rest of the agent API is fairly general and applicable to the system as a whole (at least the /authorize/* endpoints are), whereas /authorize/dekaf would be highly specific to Dekaf and nothing else.

Of course, going down the less general/more specific path has the very nice benefit of not having to worry about how to fit the things you need the API to return into the abstraction, but aren't you worried about this moving the agent API towards becoming a sort of "kitchen sink" where we add new endpoints whenever we need a new one, as opposed to presenting a more coherent interface?

I'm certainly happy to take the /authorize/dekaf path here, as like you say it'll enable returning a bunch more related data in one go, and also reduce the security-relevant surface area (as the JWT will only need access to the AVRO schemas, not all specs).

[jshearer]

Okay @jgraettinger the changes to make this into /authorize/dekaf should be done, let me know what you think.

[psFried]

LGTM

[jgraettinger]

Yep LGTM too, thanks