Cosmos Full Text Search support

[maumar]

Cosmos Full Text Search support

• Adding model building API to configure property as full-text search enabled, as well as setup the index for it,
• Adding model validation (e.g. FTS index not matching FTS property),
• Adding EF.Functions stubs and translations for FullTextContains, FullTextContainsAll, FullTextContainsAny, FullTextScore and RRF (for hybrid),
• Adding logic in SelectExpression to produce ORDER BY RANK when necessary,
• Adding validation when attempting to mix with ORDER BY RANK with regular ORDER BY,
• Rewrite OFFSET/LIMIT from parameter to constant when ORDER BY RANK is present.

Also fixed / added support for vector search on owned types (since it shares logic with FTS) and added some tests.

outstanding work:

• support for FTS Container building using Azure.ResourceManager.CosmosDb (currently blocked on updated package being released)

Fixes #35476
Fixes #35851
Fixes #35852
Fixes #35853
Fixes #35867

[Copilot]

Pull Request Overview

This PR adds support for full-text search in Cosmos by introducing new full-text index types, translators, model validations, and extensions.

• Introduces full-text indexing support in container creation and client wrappers.
• Adds a new CosmosFullTextSearchTranslator and corresponding DbFunctions extensions.
• Updates model validation and Fluent API extensions to include full-text search configuration.

Reviewed Changes

Copilot reviewed 23 out of 26 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/EFCore.Cosmos/Storage/Internal/CosmosClientWrapper.cs	Adds support for full-text indexing in container creation, but the new full-text index assignment may override vector index settings.
src/EFCore.Cosmos/Query/Internal/Translators/CosmosVectorSearchTranslator.cs	Modifies method checking logic that may prematurely reject valid method calls by changing && to
src/EFCore.Cosmos/Query/Internal/Translators/CosmosFullTextSearchTranslator.cs	Implements a new translator for full-text search functions.
src/EFCore.Cosmos/Query/Internal/Expressions/SqlFunctionExpression.cs	Introduces an extra constructor parameter to flag scoring functions.
src/EFCore.Cosmos/Query/Internal/Expressions/SelectExpression.cs	Adds a RankOrdering property and methods to apply rank-based ordering.
src/EFCore.Cosmos/Query/Internal/CosmosQueryableMethodTranslatingExpressionVisitor.cs	Adjusts ordering logic to support scoring functions and rank ordering.
src/EFCore.Cosmos/Query/Internal/CosmosQuerySqlGenerator.cs	Updates SQL generation to include rank ordering handling.
src/EFCore.Cosmos/Metadata/Internal/CosmosAnnotationNames.cs	Defines new annotations for full-text search configuration.
src/EFCore.Cosmos/Infrastructure/Internal/CosmosModelValidator.cs	Validates full-text index configuration, ensuring only single-property indexes with language annotation are defined.
src/EFCore.Cosmos/Extensions/CosmosPropertyExtensions.cs, CosmosPropertyBuilderExtensions.cs, CosmosIndexExtensions.cs, CosmosIndexBuilderExtensions.cs, CosmosDbFunctionsExtensions.cs	Adds Fluent API and extension methods for configuring full-text search on properties and indexes.

Files not reviewed (3)

• Directory.Packages.props: Language not supported
• src/EFCore.Cosmos/EFCore.Cosmos.csproj: Language not supported
• src/EFCore.Cosmos/Properties/CosmosStrings.Designer.cs: Language not supported

Comments suppressed due to low confidence (2)

src/EFCore.Cosmos/Query/Internal/Translators/CosmosVectorSearchTranslator.cs:33

• Changing the logical operator from '&&' to '||' alters the intended logic and causes the translator to return null for methods declared on CosmosDbFunctionsExtensions that are not named 'VectorDistance'. Revisit the condition to ensure it only returns null when both conditions are unmet.

if (method.DeclaringType != typeof(CosmosDbFunctionsExtensions) || method.Name != nameof(CosmosDbFunctionsExtensions.VectorDistance))

src/EFCore.Cosmos/Extensions/CosmosPropertyExtensions.cs:137

• [nitpick] The Experimental attribute for full-text search methods is using 'CosmosVectorSearchExperimental' instead of the more appropriate 'CosmosFullTextSearchExperimental'. This may cause confusion regarding the feature's experiment status.

[Experimental(EFDiagnostics.CosmosVectorSearchExperimental)]

[roji]

@maumar just to note that I'm holding back from reviewing since this is still in draft (you seemed to maybe want people to take a look in yesterday's meeting). Let me know if you want a review etc.

[maumar]

@roji feel free to hold off on reviewing a draft. I'm pretty close to marking it for review I think

[roji]

Ended up doing a quick review even though the PR is still in draft... Here are some comments.

[roji]

/cc @AndriySvyryd in case you want to take a quick look for the metadata side and Cosmos container building part.

[maumar]

@roji @AndriySvyryd the PR is now ready for review.

[roji]

@maumar see batch of comments.

I think we should maybe discuss the FTS property vs. index question in design...

[maumar]

@roji @AndriySvyryd I updated the PR addressing the Property/Index API and fixing a few small errors. You can review just the second commit - all the changes are there.

• I put the HasDefaultFullTextLanguage setting on the entity type builder, rather than creating a container builder just for it. We already have some container-level settings (e.g. TTL) and we defined them on entity type builder, so going for consistency here.
• Added tests to make sure values are propagated correctly (if property specifies language, use that, otherwise use default, otherwise use en-US). These tests need to be adjusted once cosmos supports more languages, for now they all just throw Cosmos exception complaining about not supported language. I manually verified that we wire-up things correctly when creating the container.
• Added test for using full text property without the index and removed validation preventing this scenario.
• Conditionally skipping the tests when running on live service due to Azure.ResourceManager.CosmosDb not supporting FTS.

[roji]

I still find the discrepancy between builder (IsFullTextProperty) and metadata (SetFullTextSearchLanguage) names odd and unclear. For example, what does IsFullTextProperty() actually do when a language isn't passed? Since we just call SetFullTextSearchLanguage with null, it doesn't seem to actually do anything (apart erasing a previously-set language); if so, then this method doesn't actually make the property into a full-text property - it really just sets the language.

Here's another way to look at it. Since Cosmos allows defining a default language at the container level, I'm assuming it's possible to omit it from the property level (otherwise why have a default at all). If that's the case (should confirm), then there needs to be a way to way "this is a full text property" without setting a language. How would this work with the current scheme, i.e. how does you code know to emit a container policy specifying the property as a full-text property?

If the above is true on the Cosmos side - and it needs to be confirmed by simple experimentation - we simply need to have two separate metadata: one "IsFullTextSearchEnabled" (boolean), and one "FullTextSearchLanguage" (which is optional, if not specified the container default applies). We could still of course have a single convenience builder method (I'd call it EnableFullTextSearch), which sets both metadata. According to my understand of how Cosmos actually works - that the language specification on the property level is optional - this is what we need.

One last possibility here is to use a null language to mean "this is a full text property who's language hasn't been set by the user (and should default to the container default" (is this the current design @maumar?). If so, then there's no way to turn off a property being a full-text property (since passing a null language activate FTS).

---

On another note, an alternative possible naming for the builder API could simply be EnableFullTextSearch(), since the property still remains a regular property that can be used for other things (xmldoc can also be updated to reflect that). Constrast this with vector properties, which have no other possible use.

[maumar]

One last possibility here is to use a null language to mean "this is a full text property who's language hasn't been set by the user (and should default to the container default" (is this the current design @maumar?). If so, then there's no way to turn off a property being a full-text property (since passing a null language activate FTS).

you are right, this is the current design - will switch to 2 annotations, it is much cleaner

[AndriySvyryd]

If I understood the Cosmos design correctly there is no such setting as "this is an FTS property", FTS can be used on anything by default, but to avoid specifying the language in the call it should be configured either on the container or the property. Then, if you want it to be read-efficient, you can also create an index

[AndriySvyryd]

I think this method should be called HasFullTextSearchLanguage and have a required parameter

[AndriySvyryd]

And EnableFullTextSearch() is completely unnecessary

[maumar]

@roji it is currently not allowed to set individual path's language to null or empty. SDK throws a validation error "Argument {0} can't be null or empty. (Parameter 'Language')". I'm following up with Cosmos folks to figure out exactly how it's supposed to work then, but it might be what @AndriySvyryd suggests (all properties are FTS enabled out of the box and you only specify path/language if the language is supposed to be different than the default). Documentation says something different: "For every text property you'd like to configure for full text search, you must declare both the path of the property with text and the language of the text.", so we'll see.

[maumar]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).