Single .NET direct SDK code base forward (V3+)

[kirankumarkolli]

As Cosmos echo system evolves refreshing the direct, query, CFP and cross-region availability feature sets across multiple versions is very expensive, lot more challenging (engineering cost) and error prone. Stand still with just current version (V3+) is not the end state OR an option at-least for now. V4 SDK aligning with Azure core guidelines is already on-card as next major version.

From development/engineering perspective Cosmos will be better positioned if future versions are all aligned to a single code base (preferably branch) at-least for current state direct model SDK's.

Goals

• Ability to support/deliver V3 forward along with V4
• Single branch (void porting, validations, etc...)
• Ability for support different OM heads

Support different REST API version target's (ex: V3 at v2020_07_15, V4 at v2018_12_31)
Support variable behavior changes (possibly breaking) across
Minimal common sub-set dependency in shared codebase

Big rock items along this journey

• .Net 6 specific optimizations (Transport optimizations)
• GitHub is the master of all source code (Direct, HybridRow)
• Driver/Core model: Composable mini components (HA component, query component)
• Driver/Core support multiple behaviors hot SxS behaviors
• Driver/Core: ITrace vs ClientReqeustStats abstractions
• Driver/Core minimalistic: Remove/pluggable JSON.net, HybridRow, BinaryProtocol, ConfigurationManager, Azure.Core, ITrace/Span, ServiceInterop??
• Driver OM: New OM, legacy DSR, Pruned V3 OM, Request pipeline composition. Min set: Request/Response, master OM/caches etc..
• Behaviors gates: Very well defined and codes to guard, with explicit sign-off's
• Extension components: Client encryption/SessionStateProvider, pluggable through Azure.Core or similar extension or REV each major with core Major
• OpenTelemetry for TCP connection's
• DocumentClient initialization integration into the core request pipeline (somewhat chicken and egg)
• VNext Next Major version of SDK
• LINQ and spatial types: Core or extension library? Core spatial vs Cosmos spatial
• ClientEncryption <-> Core library integration abstractions (Different from existing ones)
• Docs, samples, migration guidelines
• Supportability: Rewiring automation with V4 updated diagnostic's

Dependencies

• JIT query plan architecture for SericeInterop removal (Serice work)
• Split compute SDK
• Management place CRUD separation??
• Compute gateway SDK dependency
• V2 SDK story, if direct is forked out
• ClientEncryption to V4 funding
• Azure functions & LogcApps migration to V4 SDK
• Cosmos SessionStateProvider
• Remove HybridRow dependency

/cc: @j82w, @ealsur , @sourabh1007 , @asketagarwal , @FabianMeiswinkel, @philipthomas-microsoft, @jcocchi

[j82w]

LINQ and Spatial types should also be tracked since there is a large amount of work needed for them.

[ealsur]

Problem with Spatial types is that serialization depends on the engine. Also at some point the SDK core team did their own take on them.
Those might be something would be great to have from somewhere else (core SDK team)

[j82w]

We need to figure out how we are going to expose it and what the long term plan is. That why need to figure it out as part of the driver model. If the core version doesn't support newtonsoft then we can't use it for the v3 SDK.

[kirankumarkolli]

We need to revisit it again. Last time when Brandon looked at core types, they will not serve our scenarios. We need to revisit/re-evaluate.

[kirankumarkolli]

Reflected discussion with more context please do take a look

[JayAtSherweb]

Is there any update at all for .Net V4 SDK? This seems to have been abandoned even though the Jave V4 SDK has been available for some time now. Why are new features being added to V3 while V4 seems to have been completely abandoned? Meanwhile there are features in V4 that people have been waiting years for that are still not available in V3.

[jcocchi]

Progress on the V4 SDK is currently paused, and we will update this discussion when we have more information. Please continue to use the V3 SDK for now, and feel free to open a new issue if you're facing any problems.

[bartelink]

The reasons why Java has a V4 have little to do with this SDK.
I'd ask for specific features that address specific pain points - ultimately breaking changes are no fun for anyone, and the fragmentation from the team supporting >1 version doesnt help them or us.
What do you actually need from the V4 deliverables?

[JayAtSherweb]

It would be nice to finally have built in support for system.text.json instead of having to use a work around for this. It seems incredibly long over due at this point. V4 seemed like it was the answer to this, but then was just dropped and never bothered with again for some reason. Really really long over due. Why is cosmos still so dependent on newton soft, a third party, when Microsoft has it's own system.text.json that is used natively in Azure functions etc? It's like you're not even part of the same company or something.

[paulirwin]

Agreed with @JayAtSherweb, STJ with LINQ is what I'd like to see in v4.

[kirankumarkolli]

STJ with LINQ support is shipping in V3 shortly by end of JAN.

ref: #4138

[Davilink]

Is there any roadmap for releasing the V4 version ?

[bartelink]

This thread explains the current state #2976 (comment)

[Davilink]

@bartelink Ok, the documentation here https://learn.microsoft.com/en-us/dotnet/azure/sdk/packages for cosmos DB still point to the V4-preview, should it link to the V3 instead of the V4 ?

[bartelink]

Good catch - while its correct to say that it uses Azure.Core, the fact that it was a spike that's presently not on a roadmap means it should be removed.

Given the fact that it lists the standard supported SDK, I'd suggest that @ealsur might want to have it removed and/or at least de-emphasized/qualified...

[ealsur]

V3 does use Azure.Core and that is correct. Azure.Core contains the TokenCredential type that is used for AAD/MSI Auth.

[ealsur]

But we (Cosmos DB) do not own that page, I'm not sure how they are populating the data. V4 has been delisted from Nuget. I'll see if I can reach the folks that own it for an update. Thanks for bringing it up.