Merge branch 'ferretdb' into ci

.gitattributes

@@ -0,0 +1,4 @@
+* text=auto
+*.sh text eol=lf
+*.md text eol=lf
+*.sql text eol=lf

.github/ISSUE_TEMPLATE/bug_report.md

@@ -4,24 +4,28 @@ about: Create a report to help us improve
 title: ''
 labels: bug, new
 assignees: ''
-
 ---
 
 **Describe the bug**
+
 A clear and concise description of what the bug is.
+
 **Environment**
 - Operating system and version
 - PostgreSQL version
 - Architecture
 
 **Reproduction Steps**
+
 Steps to reproduce the behavior:
 A code snipped or accessible repository that can reproduce the issue.
 
 **Expected behavior**
+
 A clear and concise description of what you expected to happen.
 
-** Actual behavior **
+**Actual behavior**
 
 **Additional context**
+
 Add any other context about the problem here.

.github/workflows/regress_tests.yml

@@ -139,4 +139,8 @@ jobs:
 
       - name: Citus Indent
         run: |
-          citus_indent --check || (echo ""; echo "citus_indent failed, refer to CONTRIBUTING.md on how to use citus_indent"; exit 1)    
+          citus_indent --check || (echo ""; echo "citus_indent failed, refer to CONTRIBUTING.md on how to use citus_indent"; exit 1)
+
+      - name: Validate sanity of files
+        run: |
+          ./scripts/validate_extension_file_state.sh

CHANGELOG.md

@@ -0,0 +1,13 @@
+### documentdb v0.102-0 (Unreleased) ###
+* Support index pushdown for vector search queries *[Bugfix]*
+
+### documentdb v0.101-0 (February 12, 2025) ###
+* Push $graphlookup recursive CTE JOIN filters to index *[Perf]*
+* Build pg_documentdb for PostgreSQL 17 *[Infra]* (#13)
+* Enable support of currentOp aggregation stage, along with collstats, dbstats, and indexStats *[Commands]* (#52)
+* Allow inlining $unwind with $lookup with `preserveNullAndEmptyArrays` *[Perf]*
+* Skip loading documents if group expression is constant *[Perf]* 
+* Fix Merge stage not outputing to target collection *[Bugfix]* (#20)
+
+### documentdb v0.100-0 (January 23rd, 2025) ###
+Initial Release 

CODEOWNERS

@@ -2,3 +2,10 @@
 
 /.pipelines/ @safern @visridha @shuaitian-git @lichoil
 /.github/workflows/ @safern @visridha @shuaitian-git @lichoil
+
+*.csv @microsoft/documentdb-engine-reviewers
+*.sql @microsoft/documentdb-engine-reviewers
+Make* @microsoft/documentdb-engine-reviewers
+/internal/ @microsoft/documentdb-engine-reviewers
+/pg_documentdb/ @microsoft/documentdb-engine-reviewers
+/pg_documentdb_core/ @microsoft/documentdb-engine-reviewers

README.md

@@ -8,10 +8,9 @@
 
 The project comprises of two primary components, which work together to support document operations.
 
-- pg_documentdb_core : PostgreSQL extension introducing BSON datatype support and operations for native Postgres.
-- pg_documentdb : The public API surface for DocumentDB providing CRUD functionality on documents in the store.
+- **pg_documentdb_core :** PostgreSQL extension introducing BSON datatype support and operations for native Postgres.
+- **pg_documentdb :** The public API surface for DocumentDB providing CRUD functionality on documents in the store.
 
-![Architecture](images/architecture.png)
 
 ## Why DocumentDB ?
 
@@ -110,7 +109,7 @@ SELECT documentdb_api.create_collection('documentdb','patient');
 
 #### Insert documents
 
-The [documentdb_api.insertOne()](https://github.com/microsoft/documentdb/wiki/Functions#insert_one) command is used to add a single document into a collection.
+The [documentdb_api.insert_one](https://github.com/microsoft/documentdb/wiki/Functions#insert_one) command is used to add a single document into a collection.
 
 ```sql
 select documentdb_api.insert_one('documentdb','patient', '{ "patient_id": "P001", "name": "Alice Smith", "age": 30, "phone_number": "555-0123", "registration_year": "2023","conditions": ["Diabetes", "Hypertension"]}');
@@ -262,7 +261,7 @@ SELECT cursorpage FROM documentdb_api.aggregate_cursor_first_page('documentdb',
 - Please refer to page for contributing to our [Roadmap list](https://github.com/orgs/microsoft/projects/1407/views/1).
 - [FerretDB](https://github.com/FerretDB/FerretDB) integration allows using DocumentDB as backend engine.
 
-Contributors and users can join the [Discord Server](https://aka.ms/documentdb_discord) for quick collaboration.
+Contributors and users can join the [DocumentDB Discord channel in the Microsoft OSS server](https://aka.ms/documentdb_discord) for quick collaboration.
 
 ### FAQs
 

SUPPORT.md

@@ -1,25 +1,25 @@
-# TODO: The maintainer of this repo has not yet edited this file
-
-**REPO OWNER**: Do you want Customer Service & Support (CSS) support for this product/project?
-
-- **No CSS support:** Fill out this template with information about how to file issues and get help.
-- **Yes CSS support:** Fill out an intake form at [aka.ms/onboardsupport](https://aka.ms/onboardsupport). CSS will work with/help you to determine next steps.
-- **Not sure?** Fill out an intake as though the answer were "Yes". CSS will help you decide.
-
-*Then remove this first heading from this SUPPORT.MD file before publishing your repo.*
-
-# Support
-
-## How to file issues and get help  
-
-This project uses GitHub Issues to track bugs and feature requests. Please search the existing 
-issues before filing new issues to avoid duplicates.  For new issues, file your bug or 
-feature request as a new Issue.
-
-For help and questions about using this project, please **REPO MAINTAINER: INSERT INSTRUCTIONS HERE 
-FOR HOW TO ENGAGE REPO OWNERS OR COMMUNITY FOR HELP. COULD BE A STACK OVERFLOW TAG OR OTHER
-CHANNEL. WHERE WILL YOU HELP PEOPLE?**.
-
-## Microsoft Support Policy  
-
-Support for this **PROJECT or PRODUCT** is limited to the resources listed above.
+# TODO: The maintainer of this repo has not yet edited this file
+
+**REPO OWNER**: Do you want Customer Service & Support (CSS) support for this product/project?
+
+- **No CSS support:** Fill out this template with information about how to file issues and get help.
+- **Yes CSS support:** Fill out an intake form at [aka.ms/onboardsupport](https://aka.ms/onboardsupport). CSS will work with/help you to determine next steps.
+- **Not sure?** Fill out an intake as though the answer were "Yes". CSS will help you decide.
+
+*Then remove this first heading from this SUPPORT.MD file before publishing your repo.*
+
+# Support
+
+## How to file issues and get help  
+
+This project uses GitHub Issues to track bugs and feature requests. Please search the existing 
+issues before filing new issues to avoid duplicates.  For new issues, file your bug or 
+feature request as a new Issue.
+
+For help and questions about using this project, please **REPO MAINTAINER: INSERT INSTRUCTIONS HERE 
+FOR HOW TO ENGAGE REPO OWNERS OR COMMUNITY FOR HELP. COULD BE A STACK OVERFLOW TAG OR OTHER
+CHANNEL. WHERE WILL YOU HELP PEOPLE?**.
+
+## Microsoft Support Policy  
+
+Support for this **PROJECT or PRODUCT** is limited to the resources listed above.

internal/pg_documentdb_distributed/documentdb_distributed.control

@@ -1,5 +1,5 @@
 comment = 'Multi-Node API surface for DocumentDB'
-default_version = '0.100-0'
+default_version = '0.102-0'
 module_pathname = '$libdir/pg_documentdb_distributed'
 relocatable = false
 superuser = true

internal/pg_documentdb_distributed/src/distribution/distributed_hooks.c

@@ -32,6 +32,7 @@ extern bool UseLocalExecutionShardQueries;
 extern char *ApiDistributedSchemaName;
 
 extern bool EnableMetadataReferenceTableSync;
+extern char *DistributedOperationsQuery;
 
 /* Cached value for the current Global PID - can cache once
  * Since nodeId, Pid are stable.
@@ -493,6 +494,88 @@ TryGetExtendedVersionRefreshQueryCore(void)
 }
 
 
+static void
+GetShardIdsAndNamesForCollectionCore(Oid relationOid, const char *tableName,
+									 Datum **shardOidArray, Datum **shardNameArray,
+									 int32_t *shardCount)
+{
+	*shardOidArray = NULL;
+	*shardNameArray = NULL;
+	*shardCount = 0;
+	const char *query =
+		"SELECT array_agg($2 || '_' || shardid) FROM pg_dist_shard WHERE logicalrelid = $1";
+
+	int nargs = 2;
+	Oid argTypes[2] = { OIDOID, TEXTOID };
+	Datum argValues[2] = {
+		ObjectIdGetDatum(relationOid), CStringGetTextDatum(tableName)
+	};
+	bool isReadOnly = true;
+	bool isNull = true;
+	Datum shardIds = ExtensionExecuteQueryWithArgsViaSPI(query, nargs, argTypes,
+														 argValues,
+														 NULL, isReadOnly, SPI_OK_SELECT,
+														 &isNull);
+
+	if (isNull)
+	{
+		return;
+	}
+
+	ArrayType *arrayType = DatumGetArrayTypeP(shardIds);
+
+	/* Need to build the result */
+	int numItems = ArrayGetNItems(ARR_NDIM(arrayType), ARR_DIMS(arrayType));
+	Datum *resultDatums = palloc0(sizeof(Datum) * numItems);
+	Datum *resultNameDatums = palloc0(sizeof(Datum) * numItems);
+	int resultCount = 0;
+
+	const int slice_ndim = 0;
+	ArrayMetaState *mState = NULL;
+	ArrayIterator shardIterator = array_create_iterator(arrayType,
+														slice_ndim, mState);
+
+	Datum shardName = 0;
+	while (array_iterate(shardIterator, &shardName, &isNull))
+	{
+		if (isNull)
+		{
+			continue;
+		}
+
+		RangeVar *rangeVar = makeRangeVar(ApiDataSchemaName, TextDatumGetCString(
+											  shardName), -1);
+		bool missingOk = true;
+		Oid shardRelationId = RangeVarGetRelid(rangeVar, AccessShareLock, missingOk);
+		if (shardRelationId != InvalidOid)
+		{
+			Assert(resultCount < numItems);
+			resultDatums[resultCount] = shardRelationId;
+			resultNameDatums[resultCount] = PointerGetDatum(DatumGetTextPCopy(
+																shardName));
+			resultCount++;
+		}
+	}
+
+	array_free_iterator(shardIterator);
+
+	/* Now that we have the shard list as a Datum*, create an array type */
+	if (resultCount > 0)
+	{
+		*shardOidArray = resultDatums;
+		*shardNameArray = resultNameDatums;
+		*shardCount = resultCount;
+	}
+	else
+	{
+		pfree(resultDatums);
+		pfree(resultNameDatums);
+	}
+
+	pfree(arrayType);
+}
+
+
 /*
  * Register hook overrides for DocumentDB.
  */
@@ -516,4 +599,8 @@ InitializeDocumentDBDistributedHooks(void)
 	UpdateColocationHooks();
 
 	try_get_extended_version_refresh_query_hook = TryGetExtendedVersionRefreshQueryCore;
+	get_shard_ids_and_names_for_collection_hook = GetShardIdsAndNamesForCollectionCore;
+
+	DistributedOperationsQuery =
+		"SELECT * FROM pg_stat_activity LEFT JOIN pg_catalog.get_all_active_transactions() ON process_id = pid";
 }

internal/pg_documentdb_distributed/src/test/regress/basic_schedule_core

@@ -85,7 +85,7 @@ test: bson_get_indexes_a commands_create_indexes_pfe_a commands_crud_ignore_comm
 test: write_commands_batching conflicting_indexes
 
 test: bson_aggregation_pipeline_tests_vector_mongo bson_aggregation_expression_variable_tests bson_aggregation_pipeline_tests_vector_hnsw
-test: bson_aggregation_pipeline_tests_vector_ivf list_metadata_cursor_tests
+test: bson_aggregation_pipeline_tests_vector_ivf list_metadata_cursor_tests bson_aggregation_pipeline_tests_vector_hnsw_planner
 test: schema_validation_insert schema_validation
 test: bson_aggregation_pipeline_operator_locf commands_insert setwindowfields_and_group_compliance
 

internal/pg_documentdb_distributed/src/test/regress/bin/expect_normalize.sed

@@ -8,4 +8,6 @@
 
 
 # Differing names can have differing table column widths
-s/^-[+-]{2,}$/---------------------------------------------------------------------/g
+s/^-[+-]{2,}$/---------------------------------------------------------------------/g
+# Replace the values of the time system variables ($$NOW) with a constant
+s/\"sn\" : \{ \"\$date\" : \{ \"\$numberLong\" : \"[0-9]*\" \} \}/\"sn\" : NOW_SYS_VARIABLE/g

internal/pg_documentdb_distributed/src/test/regress/bin/normalize.sed

@@ -18,4 +18,7 @@ s/process [0-9]+ acquired ShareLock on transaction [0-9]+ after [0-9\.]+ ms/proc
 s/Distributed Subplan \d+/Distributed Subplan DDD/g
 s/Distributed Subplan [0-9]+_[0-9]+/Distributed Subplan X_X/g
 s/read_intermediate_result\('[0-9]+_[0-9]+'::text/read_intermediate_result\('X_X'::text/g
-s/Type oid not supported \d+/Type oid not supported ddd/g
+s/Type oid not supported \d+/Type oid not supported ddd/g
+# Replace the values of the $$NOW time system variable with a constant
+s/\"now\" : \{ \"\$date\" : \{ \"\$numberLong\" : \"[0-9]*\" \} \}/\"now\" : NOW_SYS_VARIABLE/g
+s/\"sn\" : \{ \"\$date\" : \{ \"\$numberLong\" : \"[0-9]*\" \} \}/\"sn\" : NOW_SYS_VARIABLE/g