diff --git a/source/includes/copy-existing-admonition.rst b/source/includes/copy-existing-admonition.rst index 835ac27..1c51d14 100644 --- a/source/includes/copy-existing-admonition.rst +++ b/source/includes/copy-existing-admonition.rst @@ -1,7 +1,5 @@ -.. note:: Data Copy Can Produce Duplicate Events - - If any system changes the data in the database while the source connector - converts existing data from it, MongoDB may produce duplicate change - stream events to reflect the latest changes. Since the change stream - events on which the data copy relies are idempotent, the copied data is - eventually consistent. +If any system changes the data in the database while the source connector +converts existing data from it, MongoDB may produce duplicate change +stream events to reflect the latest changes. Since the change stream +events on which the data copy relies are idempotent, the copied data is +eventually consistent. diff --git a/source/includes/externalize-secrets.rst b/source/includes/externalize-secrets.rst index 3d8054a..749a60b 100644 --- a/source/includes/externalize-secrets.rst +++ b/source/includes/externalize-secrets.rst @@ -1,7 +1,5 @@ -.. important:: Avoid Exposing Your Authentication Credentials - - To avoid exposing your authentication credentials in your - ``connection.uri`` setting, use a - `ConfigProvider `__ - and set the appropriate configuration parameters. +:gold:`IMPORTANT:` To avoid exposing your authentication credentials in your +``connection.uri`` setting, use a +`ConfigProvider `__ +and set the appropriate configuration parameters. diff --git a/source/introduction/data-formats.txt b/source/introduction/data-formats.txt index e6baf41..fd447f1 100644 --- a/source/introduction/data-formats.txt +++ b/source/introduction/data-formats.txt @@ -1,3 +1,5 @@ +.. _kafka-data-formats: + ============ Data Formats ============ diff --git a/source/sink-connector/configuration-properties/connector-message.txt b/source/sink-connector/configuration-properties/connector-message.txt index 06a988f..93db2e3 100644 --- a/source/sink-connector/configuration-properties/connector-message.txt +++ b/source/sink-connector/configuration-properties/connector-message.txt @@ -75,8 +75,9 @@ Settings When set to ``true``, the default value, the connector writes a batch of records as an ordered bulk write operation. - | To learn more about bulk write operations, see the :ref:`Write - Model Strategies page `. + | + | To learn more about bulk write operations, see + :ref:`Bulk Write Operations `. | | **Default**: ``true`` | **Accepted Values**: ``true`` or ``false`` @@ -111,15 +112,13 @@ Settings | The maximum number of tasks to create for this connector. The connector may create fewer than the maximum tasks specified if it cannot handle the level of parallelism you specify. - - .. important:: Multiple Tasks May Process Messages Out of Order - - If you specify a value greater than ``1``, the connector enables - parallel processing of the tasks. If your topic has multiple - partition logs, which enables the connector to read from the - topic in parallel, the tasks may process the messages out of - order. - + | + | :gold:`IMPORTANT:` If you specify a value greater than ``1``, + the connector enables parallel processing of the tasks. + If your topic has multiple partition logs, which enables + the connector to read from the topic in parallel, + the tasks may process the messages out of order. + | | **Default**: ``1`` | **Accepted Values**: An integer diff --git a/source/sink-connector/configuration-properties/error-handling.txt b/source/sink-connector/configuration-properties/error-handling.txt index 4fbf2b9..c29450c 100644 --- a/source/sink-connector/configuration-properties/error-handling.txt +++ b/source/sink-connector/configuration-properties/error-handling.txt @@ -48,19 +48,21 @@ Settings | Whether to continue processing messages if the connector encounters an error. Allows the connector to override the ``errors.tolerance`` Kafka cluster setting. + | | When set to ``none``, the connector reports any error and blocks further processing of the rest of the messages. + | | When set to ``all``, the connector ignores any problematic messages. + | | When set to ``data``, the connector tolerates only data errors and fails on all other errors. + | | To learn more about error handling strategies, see the :ref:`` page. - - .. note:: - - This property overrides the `errors.tolerance `__ - property of the Connect Framework. - + | + | This property overrides the `errors.tolerance `__ + | property of the Connect Framework. + | | **Default:** Inherits the value from the ``errors.tolerance`` setting. | **Accepted Values**: ``"none"`` or ``"all"`` @@ -73,16 +75,14 @@ Settings failed operations to the log file. The connector classifies errors as "tolerated" or "not tolerated" using the ``errors.tolerance`` or ``mongo.errors.tolerance`` settings. - + | | When set to ``true``, the connector logs both "tolerated" and "not tolerated" errors. | When set to ``false``, the connector logs "not tolerated" errors. - - .. note:: - - This property overrides the `errors.log.enable `__ - property of the Connect Framework. - + | + | This property overrides the `errors.log.enable `__ + | property of the Connect Framework. + | | **Default:** ``false`` | **Accepted Values**: ``true`` or ``false`` @@ -104,7 +104,8 @@ Settings | Name of topic to use as the dead letter queue. If blank, the connector does not send any invalid messages to the dead letter queue. - | For more information about the dead letter queue, see the + | + | To learn more about the dead letter queue, see the :ref:`Dead Letter Queue Configuration Example `. | | **Default:** ``""`` @@ -116,11 +117,13 @@ Settings | **Description:** | Whether the connector should include context headers when it writes messages to the dead letter queue. + | | To learn more about the dead letter queue, see the :ref:`Dead Letter Queue Configuration Example `. + | | To learn about the exceptions the connector defines and - reports through context headers, see the - :ref:`` section. + reports through context headers, see + :ref:``. | | **Default:** ``false`` | **Accepted Values**: ``true`` or ``false`` @@ -132,7 +135,8 @@ Settings | The number of nodes on which to replicate the dead letter queue topic. If you are running a single-node Kafka cluster, you must set this to ``1``. - | For more information about the dead letter queue, see the + | + | To learn more about the dead letter queue, see the :ref:`Dead Letter Queue Configuration Example `. | | **Default:** ``3`` diff --git a/source/sink-connector/configuration-properties/id-strategy.txt b/source/sink-connector/configuration-properties/id-strategy.txt index 5b0fba4..1a684b6 100644 --- a/source/sink-connector/configuration-properties/id-strategy.txt +++ b/source/sink-connector/configuration-properties/id-strategy.txt @@ -76,8 +76,9 @@ Settings | | **Description:** | Whether the connector should delete documents when the key value - matches a document in MongoDB and the value field is null. This - setting applies when you specify an id generation strategy that + matches a document in MongoDB and the value field is null. + | + | This setting applies when you specify an id generation strategy that operates on the key document such as ``FullKeyStrategy``, ``PartialKeyStrategy``, and ``ProvidedInKeyStrategy``. | diff --git a/source/sink-connector/configuration-properties/kafka-topic.txt b/source/sink-connector/configuration-properties/kafka-topic.txt index ef4777f..ee05b89 100644 --- a/source/sink-connector/configuration-properties/kafka-topic.txt +++ b/source/sink-connector/configuration-properties/kafka-topic.txt @@ -42,10 +42,8 @@ Settings | **Description:** | A list of Kafka topics that the sink connector watches. - .. note:: - - You can define either the ``topics`` or the ``topics.regex`` - setting, but not both. + You can define either the ``topics`` or the ``topics.regex`` + setting, but not both. | **Accepted Values**: A comma-separated list of valid Kafka topics @@ -58,20 +56,16 @@ Settings | A regular expression that matches the Kafka topics that the sink connector watches. - .. example:: - - .. code-block:: properties - - topics.regex=activity\\.\\w+\\.clicks$ + For example, the following regex matches topic names such as + "activity.landing.clicks" and "activity.support.clicks". + It does not match the topic names "activity.landing.views" and "activity.clicks". - This regex matches topic names such as "activity.landing.clicks" - and "activity.support.clicks". It does not match the topic names - "activity.landing.views" and "activity.clicks". + .. code-block:: properties - .. note:: + topics.regex=activity\\.\\w+\\.clicks$ - You can define either the ``topics`` or the ``topics.regex`` - setting, but not both. + You can define either the ``topics`` or the ``topics.regex`` + setting, but not both. | **Accepted Values**: A valid regular expression pattern using ``java.util.regex.Pattern``. diff --git a/source/sink-connector/configuration-properties/mongodb-connection.txt b/source/sink-connector/configuration-properties/mongodb-connection.txt index 73a412f..4b8dcf9 100644 --- a/source/sink-connector/configuration-properties/mongodb-connection.txt +++ b/source/sink-connector/configuration-properties/mongodb-connection.txt @@ -68,6 +68,7 @@ Settings | When set to ``true``, if the connector calls a command on your MongoDB instance that's deprecated in the declared {+stable-api+} version, it raises an exception. + | | You can set the API version with the ``server.api.version`` configuration option. For more information on the {+stable-api+}, see the MongoDB manual entry on the @@ -83,6 +84,7 @@ Settings | When set to ``true``, if the connector calls a command on your MongoDB instance that's not covered in the declared {+stable-api+} version, it raises an exception. + | | You can set the API version with the ``server.api.version`` configuration option. For more information on the {+stable-api+}, see the MongoDB manual entry on the diff --git a/source/sink-connector/configuration-properties/mongodb-namespace.txt b/source/sink-connector/configuration-properties/mongodb-namespace.txt index 47fb750..964053e 100644 --- a/source/sink-connector/configuration-properties/mongodb-namespace.txt +++ b/source/sink-connector/configuration-properties/mongodb-namespace.txt @@ -43,14 +43,12 @@ Settings database or collection in which to sink the data. The default ``DefaultNamespaceMapper`` uses values specified in the ``database`` and ``collection`` properties. - - .. seealso:: - - The connector includes an alternative class for specifying the - database and collection called ``FieldPathNamespaceMapper``. See - the :ref:`FieldPathNamespaceMapper settings ` - for more information. - + | + | The connector includes an alternative class for specifying the + | database and collection called ``FieldPathNamespaceMapper``. See + | :ref:`FieldPathNamespaceMapper Settings ` + | for more information. + | | **Default**: .. code-block:: none @@ -149,10 +147,12 @@ You can use the following settings to customize the behavior of the | **Description**: | Whether to throw an exception when either the document is missing the mapped field or it has an invalid BSON type. + | | When set to ``true``, the connector does not process documents missing the mapped field or that contain an invalid BSON type. The connector may halt or skip processing depending on the related error-handling configuration settings. + | | When set to ``false``, if a document is missing the mapped field or if it has an invalid BSON type, the connector defaults to writing to the specified ``database`` and ``collection`` settings. diff --git a/source/sink-connector/configuration-properties/post-processors.txt b/source/sink-connector/configuration-properties/post-processors.txt index f5c512a..0d89b21 100644 --- a/source/sink-connector/configuration-properties/post-processors.txt +++ b/source/sink-connector/configuration-properties/post-processors.txt @@ -40,13 +40,10 @@ Settings | **Description:** | A list of post-processor classes the connector should apply to process the data before saving it to MongoDB. - - .. seealso:: - - For more information on post-processors and examples of - their usage, see the section on - :doc:`Post-processors `. - + | + | To learn more about post-processors and see examples of + | their usage, see + | :doc:`Sink Connector Post Processors `. | | **Default**: @@ -129,12 +126,9 @@ Settings | **Description:** | The class that specifies the ``WriteModelStrategy`` the connector should use for :manual:`Bulk Writes `. - - .. seealso:: - - For information on how to create your own strategy, see - :ref:``. - + | + | To learn more about how to create your own strategy, see + | :ref:``. | | **Default**: diff --git a/source/sink-connector/configuration-properties/time-series.txt b/source/sink-connector/configuration-properties/time-series.txt index e34506c..ae6c9ff 100644 --- a/source/sink-connector/configuration-properties/time-series.txt +++ b/source/sink-connector/configuration-properties/time-series.txt @@ -57,9 +57,11 @@ Settings | The date format pattern the connector should use to convert the source data contained in the field specified by the ``timeseries.timefield`` setting. - The connector passes the date format pattern to the Java + | + | The connector passes the date format pattern to the Java `DateTimeFormatter.ofPattern(pattern, locale) `__ method to perform date and time conversions on the time field. + | | If the date value from the source data only contains date information, the connector sets the time information to the start of the specified day. If the date value does not contain the timezone offset, the @@ -79,6 +81,7 @@ Settings | **Description:** | Whether to convert the data in the field into the BSON ``Date`` format. + | | When set to ``true``, the connector uses the milliseconds after epoch and discards fractional parts if the value is a number. If the value is a string, the connector uses the @@ -99,8 +102,9 @@ Settings | | **Description:** | Which ``DateTimeFormatter`` locale language tag to use with the date - format pattern (e.g. ``"en-US"``). For more information on - locales, see the Java SE documentation of `Locale `__. + format pattern (e.g. ``"en-US"``). + | + | To learn more about locales, see the Java SE documentation of `Locale `__. | | **Default**: ``ROOT`` | **Accepted Values**: A valid ``Locale`` language tag format @@ -111,12 +115,10 @@ Settings | **Description:** | Which top-level field to read from the source data to describe a group of related time series documents. - - .. important:: - - This field must not be the ``_id`` field nor the field you specified - in the ``timeseries.timefield`` setting. - + | + | :gold:`IMPORTANT:` This field must not be the ``_id`` field nor the field you specified + in the ``timeseries.timefield`` setting. + | | **Default**: ``""`` | **Accepted Values**: An empty string or the name of a field that contains any BSON type except ``BsonArray``. @@ -128,8 +130,9 @@ Settings | The number of seconds MongoDB should wait before automatically removing the time series collection data. The connector disables timed expiry when the setting value is less than ``1``. - For more information on this collection setting, see the MongoDB - Server Manual page on :manual:`Automatic Removal for Time Series Collections `. + | + | To learn more, see :manual:`Set up Automatic Removal for Time Series Collections ` + in the MongoDB manual. | | **Default**: ``0`` | **Accepted Values**: An integer @@ -140,9 +143,10 @@ Settings | | **Description:** | The expected interval between subsequent measurements of your - source data. For more information on this setting, see the - MongoDB server manual page on :manual:`Granularity for Time - Series Data `. + source data. + | + | To learn more, see :manual:`Set Granularity for Time Series Data + ` in the MongoDB manual. | | *Optional* | **Default**: ``""`` diff --git a/source/sink-connector/configuration-properties/topic-override.txt b/source/sink-connector/configuration-properties/topic-override.txt index a592776..d6d39f9 100644 --- a/source/sink-connector/configuration-properties/topic-override.txt +++ b/source/sink-connector/configuration-properties/topic-override.txt @@ -40,19 +40,15 @@ Settings | **Description:** | Specify a topic and property name to override the corresponding global or default property setting. - - .. example:: - - The ``topic.override.foo.collection=bar`` setting instructs the - sink connector to store data from the ``foo`` topic in the ``bar`` - collection. - - .. note:: - - You can specify any valid configuration setting in the - ```` segment on a per-topic basis except - ``connection.uri`` and ``topics``. - + | + | For example, the ``topic.override.foo.collection=bar`` setting instructs + | the sink connector to store data from the ``foo`` topic in the ``bar`` + | collection. + | + | You can specify any valid configuration setting in the + | ```` segment on a per-topic basis except + | ``connection.uri`` and ``topics``. + | | **Default**: ``""`` | **Accepted Values**: Accepted values specific to the overridden property diff --git a/source/source-connector/configuration-properties/change-stream.txt b/source/source-connector/configuration-properties/change-stream.txt index 925de1c..2c4b810 100644 --- a/source/source-connector/configuration-properties/change-stream.txt +++ b/source/source-connector/configuration-properties/change-stream.txt @@ -44,18 +44,18 @@ Settings | An array of aggregation pipelines to run in your change stream. You must configure this setting for the change stream event document, not the ``fullDocument`` field. + | + | For example: - .. example:: + .. code-block:: none - .. code-block:: none + [{"$match": { "$and": [{"operationType": "insert"}, {"fullDocument.eventId": 1404 }] } }] - [{"$match": { "$and": [{"operationType": "insert"}, {"fullDocument.eventId": 1404 }] } }] - - .. tip:: Additional Examples - - - :ref:`` - - :ref:`` + | For more examples, see: + - :ref:`` + - :ref:`` + | **Default**: ``"[]"`` | **Accepted Values**: Valid aggregation pipeline stage @@ -65,22 +65,24 @@ Settings | **Description:** | Determines what values your change stream returns on update operations. + | | The default setting returns the differences between the original document and the updated document. + | | The ``updateLookup`` setting returns the differences between the original document and updated document as well as a copy of the entire updated document at a *point in time* after the update. + | | The ``whenAvailable`` setting returns the updated document, if available. + | | The ``required`` setting returns the updated document and raises an error if it is not available. - - .. tip:: - - For more information on how this change stream option works, see - the MongoDB server manual guide on :manual:`Lookup Full Document for - Update Operations `. - + | + | For more information on how this change stream option works, see + :manual:`Lookup Full Document for Update Operations + ` + in the MongoDB manual. | | **Default**: ``""`` | **Accepted Values**: ``""``, ``"updateLookup"``, ``"whenAvailable"``, or ``"required"`` @@ -93,17 +95,17 @@ Settings operations. The pre-image is not available for source records published while copying existing data, and the pre-image configuration has no effect on copying. - - .. tip:: - - To learn how to configure a collection to enable - pre-images, see the :manual:`Server manual entry on pre- and - post-images `. - + | + | To learn how to configure a collection to enable + pre-images, see :manual:`Change Streams with Document Pre- and Post-Images + ` in the MongoDB manual. + | | The default setting suppresses the document pre-image. + | | The ``whenAvailable`` setting returns the document pre-image if it's available, before it was replaced, updated, or deleted. + | | The ``required`` setting returns the document pre-image and raises an error if it is not available. | @@ -120,12 +122,12 @@ Settings updated document. To learn more about the ``fullDocument`` field, see the :manual:`update Event ` in the Server manual. - + | | When set to ``true``, the connector overrides the ``change.stream.full.document`` setting and sets it to ``updateLookup`` so that the ``fullDocument`` field contains updated documents. - + | | **Default**: ``false`` | **Accepted Values**: ``true`` or ``false`` @@ -137,7 +139,7 @@ Settings Tombstone events contain the keys of deleted documents with ``null`` values. This setting applies only when ``publish.full.document.only`` is ``true``. - + | | **Default**: ``false`` | **Accepted Values**: ``true`` or ``false`` @@ -147,12 +149,12 @@ Settings | **Description:** | Whether to use the document key for the source record key if the document key is present. - + | | When set to ``true``, the connector adds keys of the deleted documents to the tombstone events. When set to ``false``, the connector uses the resume token as the source key for the tombstone events. - + | | **Default**: ``true`` | **Accepted Values**: ``true`` or ``false`` diff --git a/source/source-connector/configuration-properties/error-handling.txt b/source/source-connector/configuration-properties/error-handling.txt index 1a5b0f1..9a8172e 100644 --- a/source/source-connector/configuration-properties/error-handling.txt +++ b/source/source-connector/configuration-properties/error-handling.txt @@ -39,17 +39,17 @@ Settings | **Description:** | Whether to continue processing messages when the connector encounters an error. + | | Set this to ``"none"`` if you want the connector to stop processing messages and report the issue if it encounters an error. + | | Set this to ``"all"`` if you want the connector to continue processing messages and ignore any errors it encounters. - - .. important:: - - This property overrides the `errors.tolerance `__ - Connect Framework property. - + | + | :gold:`IMPORTANT:` This property overrides the + `errors.tolerance `__ + Connect Framework property. | | **Default:** ``"none"`` | **Accepted Values**: ``"none"`` or ``"all"`` @@ -59,17 +59,17 @@ Settings | | **Description:** | Whether the connector should report errors in the log file. + | | Set this to ``true`` to log all errors the connector encounters. + | | Set this to ``false`` to log errors that are not tolerated by the connector. You can specify which errors the connector should tolerate using the ``errors.tolerance`` or ``mongo.errors.tolerance`` setting. - - .. important:: - - This property overrides the `errors.log.enable `__ - Connect Framework property. - + | + | :gold:`IMPORTANT:` This property overrides the + `errors.log.enable `__ + Connect Framework property. | | **Default:** ``false`` | **Accepted Values**: ``true`` or ``false`` @@ -79,16 +79,16 @@ Settings | | **Description:** | The name of topic to use as the dead letter queue. + | | If you specify a value, the connector writes invalid messages to the dead letter queue topic as :manual:`extended JSON strings `. + | | If you leave this setting blank, the connector does not write invalid messages to any topic. - .. important:: - - You must set ``errors.tolerance`` or ``mongo.errors.tolerance`` - setting to ``"all"`` to enable this property. - + | :gold:`IMPORTANT:` You must set ``errors.tolerance`` or ``mongo.errors.tolerance`` + setting to ``"all"`` to enable this property. + | | **Default:** ``""`` | **Accepted Values**: A valid Kafka topic name @@ -98,12 +98,14 @@ Settings | **Description:** | The custom offset partition name to use. You can use this option to instruct the connector to start a new change stream when an - existing offset contains an invalid resume token. If you leave - this setting blank, the connector uses the default partition name - based on the connection details. To view a strategy for naming - offset partitions, see the :ref:`` - guide. - + existing offset contains an invalid resume token. + | + | If you leave this setting blank, the connector uses the default partition name + based on the connection details. + | + | To view a strategy for naming + offset partitions, see :ref:``. + | | **Default:** ``""`` | **Accepted Values**: A string. To learn more about naming a partition, see @@ -116,15 +118,19 @@ Settings | **Description:** | The number of milliseconds the connector waits between sending heartbeat messages. The connector sends heartbeat messages when - source records are not published in the specified interval. + source records are not published in the specified interval. This mechanism improves + resumability of the connector for low volume namespaces. + | | Heartbeat messages contain a ``postBatchResumeToken`` data field. The value of this field contains the MongoDB server oplog entry that the connector last read from the change stream. - | This mechanism improves resumability of the connector for low volume - namespaces. See the :ref:`Invalid Resume Token ` - page for more information on this feature. - + | | Set this to ``0`` to disable heartbeat messages. + | + | To learn more, see :ref:`Prevention ` + in the :ref:`Invalid Resume Token ` + page. + | | **Default**: ``0`` | **Accepted Values**: An integer diff --git a/source/source-connector/configuration-properties/kafka-topic.txt b/source/source-connector/configuration-properties/kafka-topic.txt index 61b9d38..e95444d 100644 --- a/source/source-connector/configuration-properties/kafka-topic.txt +++ b/source/source-connector/configuration-properties/kafka-topic.txt @@ -50,12 +50,9 @@ Settings The destination topic name is composed of the ``topic.prefix`` value followed by the database and collection names, separated by the value specified in the ``topic.separator`` property. - - .. seealso:: - - :ref:`Topic Naming Prefix ` - usage example - + | + | To learn more, see the example in :ref:`Topic Naming Prefix `. + | | **Default**: ``""`` | **Accepted Values**: A string composed of ASCII alphanumeric characters including ".", "-", and "_" @@ -69,12 +66,9 @@ Settings The destination topic name is composed of the database and collection names followed by the ``topic.suffix`` value, separated by the value specified in the ``topic.separator`` property. - - .. seealso:: - - :ref:`Topic Naming Suffix ` - usage example - + | + | To learn more, see the example in :ref:`Topic Naming Suffix `. + | | **Default**: ``""`` | **Accepted Values**: A string composed of ASCII alphanumeric characters including ".", "-", and "_" @@ -86,16 +80,15 @@ Settings | Specifies a JSON mapping between change stream document :manual:`namespaces ` and topic names. - - .. tip:: Namespace Mapping Behavior - - You can specify complex mappings by using the - ``topic.namespace.map`` property. This property supports regex - and wildcard matching. To learn more about these behaviors and - view examples, see the :ref:`Topic Namespace Map - ` section of the Topic Naming - page. - + | + | You can use to ``topic.namespace.map`` property to + specify complex mappings. This property supports regex + and wildcard matching. + | + | To learn more about these behaviors and + view examples, see :ref:`Topic Namespace Map + `. + | | **Default**: ``""`` | **Accepted Values**: A valid JSON object @@ -113,30 +106,24 @@ Settings #. ``collection`` #. ``topic.suffix`` - .. example:: + | For example, the following configuration instructs the connector to publish + change stream documents from the ``coll`` collection of the + ``db`` database to the ``prefix-db-coll`` topic: - The following configuration instructs the connector to publish - change stream documents from the ``coll`` collection of the - ``db`` database to the ``prefix-db-coll`` - topic: + .. code-block:: properties + :copyable: false - .. code-block:: properties - :copyable: false - - topic.prefix=prefix - database=db - collection=coll - topic.separator=- - - .. important:: Topic Separator and Topic Namespace Map - - When you use the ``topic.separator`` property, keep in mind that it - does not affect how you define the ``topic.namespace.map`` property. - The ``topic.namespace.map`` property uses MongoDB - :manual:`namespaces ` - which you must always specify with a ``"."`` character to separate - the database and collection name. + topic.prefix=prefix + database=db + collection=coll + topic.separator=- + | :gold:`IMPORTANT:` When you use the ``topic.separator`` property, note that it + doesn't affect how you define the ``topic.namespace.map`` property. + The ``topic.namespace.map`` property uses MongoDB + :manual:`namespaces ` + which you must always specify with a ``.`` character to separate + the database and collection name. | | **Default**: ``"."`` | **Accepted Values**: A string diff --git a/source/source-connector/configuration-properties/mongodb-connection.txt b/source/source-connector/configuration-properties/mongodb-connection.txt index e84032a..9d01c04 100644 --- a/source/source-connector/configuration-properties/mongodb-connection.txt +++ b/source/source-connector/configuration-properties/mongodb-connection.txt @@ -49,7 +49,8 @@ Settings | **Description:** | The :manual:`URI connection string ` to connect to your MongoDB instance or cluster. - | For more information, see the :ref:`` guide + | + | To learn more, see :ref:``. .. include:: /includes/externalize-secrets.rst @@ -72,12 +73,9 @@ Settings | **Description:** | Name of the collection in the database to watch for changes. If not set, the connector watches all collections for changes. - - .. important:: - - If your ``database`` configuration is set to ``""``, the connector - ignores the ``collection`` setting. - + | + | :gold:`IMPORTANT:` If your ``database`` configuration is set to ``""``, the connector + ignores the ``collection`` setting. | | **Default**: ``""`` | **Accepted Values**: A single collection name @@ -101,6 +99,7 @@ Settings | When set to ``true``, if the connector calls a command on your MongoDB instance that's deprecated in the declared {+stable-api+} version, it raises an exception. + | | You can set the API version with the ``server.api.version`` configuration option. For more information on the {+stable-api+}, see the MongoDB manual entry on the @@ -116,6 +115,7 @@ Settings | When set to ``true``, if the connector calls a command on your MongoDB instance that's not covered in the declared {+stable-api+} version, it raises an exception. + | | You can set the API version with the ``server.api.version`` configuration option. For more information on the {+stable-api+}, see the MongoDB manual entry on the diff --git a/source/source-connector/configuration-properties/output-format.txt b/source/source-connector/configuration-properties/output-format.txt index a2a6361..1412966 100644 --- a/source/source-connector/configuration-properties/output-format.txt +++ b/source/source-connector/configuration-properties/output-format.txt @@ -51,17 +51,15 @@ Settings | Specifies which data format the source connector outputs the value document. | + | The connector supports Protobuf as an + output data format. You can enable this format by specifying the + ``schema`` value and installing and :ref:`configuring + ` the `Kafka Connect Protobuf + Converter `__. + | | **Default**: ``json`` | **Accepted Values**: ``bson``, ``json``, ``schema`` - .. note:: Protobuf Output Format Support - - The connector supports Protobuf as an - output data format. You can enable this format by specifying the - ``schema`` value and installing and :ref:`configuring - ` the `Kafka Connect Protobuf - Converter `__. - * - | **output.json.formatter** - | **Type:** string | @@ -91,14 +89,12 @@ Settings - | **Type:** string | | **Description:** - | Specifies an AVRO schema definition for the key document of the + | Specifies an Avro schema definition for the key document of the `SourceRecord <{+kafka_api_docs_base+}javadoc/org/apache/kafka/connect/source/SourceRecord.html>`__. - - .. seealso:: - - For more information on AVRO schema, see the - :ref:`Data Formats ` guide. - + | + | To learn more about Avro schema, see :ref:`Avro ` in the + :ref:`Data Formats guide `. + | | **Default**: .. code-block:: json @@ -109,20 +105,18 @@ Settings "fields" : [ { "name": "_id", "type": "string" } ]" } - | **Accepted Values**: A valid AVRO schema + | **Accepted Values**: A valid Avro schema * - | **output.schema.value** - | **Type:** string | | **Description:** - | Specifies an AVRO schema definition for the value document of the + | Specifies an Avro schema definition for the value document of the `SourceRecord <{+kafka_api_docs_base+}javadoc/org/apache/kafka/connect/source/SourceRecord.html>`__. - - .. seealso:: - - For more information on AVRO schema, see the - :ref:`Data Formats ` guide. - + | + | To learn more about Avro schema, see :ref:`Avro ` in the + :ref:`Data Formats guide `. + | | **Default**: .. code-block:: json @@ -169,12 +163,10 @@ Settings document of the `SourceRecord <{+kafka_api_docs_base+}javadoc/org/apache/kafka/connect/source/SourceRecord.html>`__. Since the connector processes each document in isolation, the connector may generate many schemas. - - .. important:: - - The connector only reads this setting when you set your - ``output.format.value`` setting to ``schema``. - + | + | :gold:`IMPORTANT:` The connector only reads this setting when you set your + ``output.format.value`` setting to ``schema``. + | | **Default**: ``false`` | **Accepted Values**: ``true`` or ``false`` diff --git a/source/source-connector/configuration-properties/startup.txt b/source/source-connector/configuration-properties/startup.txt index b5cef31..359fa07 100644 --- a/source/source-connector/configuration-properties/startup.txt +++ b/source/source-connector/configuration-properties/startup.txt @@ -65,7 +65,7 @@ Settings setting is equivalent to the deprecated setting ``copy.existing=true``. .. include:: /includes/copy-existing-admonition.rst - + | **Default**:``latest`` | **Accepted Values**: ``latest``, ``timestamp``, ``copy_existing`` @@ -74,9 +74,12 @@ Settings | | **Description:** | Actuated only if ``startup.mode=timestamp``. Specifies the - starting point for the change stream. To learn more about - Change Stream parameters, see the :manual:`Server manual entry - `. + starting point for the change stream. + | + | To learn more about Change Stream parameters, see + :manual:`$changeStream (aggregation) ` + in the MongoDB manual. + | | **Default**: ``""`` | **Accepted Values**: @@ -94,21 +97,20 @@ Settings | Regular expression the connector uses to match namespaces from which to copy data. A namespace describes the MongoDB database name and collection separated by a period (for example, ``databaseName.collectionName``). + | + | For example, the following regular-expression setting matches + collections that start with "page" in the ``stats`` database: - .. example:: - - In the following example, the regular-expression setting matches - collections that start with "page" in the ``stats`` database. - - .. code-block:: none - - startup.mode.copy.existing.namespace.regex=stats\.page.* + .. code-block:: none - The "\" character in the example above escapes the "." character - that follows it in the regular expression. For more information on - how to build regular expressions, see the Java API documentation on - `Patterns `__. + startup.mode.copy.existing.namespace.regex=stats\.page.* + | The ``\`` character in the example above escapes the ``.`` character + that follows it in the regular expression. For more information on + how to build regular expressions, see + `Patterns `__ + in the Java API documentation. + | | **Default**: ``""`` | **Accepted Values**: A valid regular expression @@ -120,16 +122,14 @@ Settings the connector runs when copying existing data. You can use this setting to filter the source collection and improve the use of indexes in the copying process. + | + | For example, the following setting uses the :manual:`$match ` + aggregation operator to instruct the connector to copy only + documents that contain a ``closed`` field with a value of ``false``. - .. example:: - - The following example shows how you can use the :manual:`$match ` - aggregation operator to instruct the connector to copy only - documents that contain a ``closed`` field with a value of ``false``. - - .. code-block:: none + .. code-block:: none - startup.mode.copy.existing.pipeline=[ { "$match": { "closed": "false" } } ] + startup.mode.copy.existing.pipeline=[ { "$match": { "closed": "false" } } ] | **Default**: ``""`` | **Accepted Values**: Valid aggregation pipeline stages @@ -139,6 +139,7 @@ Settings | | **Description:** | The maximum number of threads the connector can use to copy data. + | | **Default**: number of processors available in the environment | **Accepted Values**: An integer @@ -147,6 +148,7 @@ Settings | | **Description:** | The size of the queue the connector can use when copying data. + | | **Default**: ``16000`` | **Accepted Values**: An integer @@ -156,6 +158,7 @@ Settings | **Description:** | When set to ``true``, the connector uses temporary disk storage for the copy existing aggregation. + | | **Default**: ``true`` | **Accepted Values**: ``true`` or ``false`` diff --git a/source/source-connector/usage-examples/copy-existing-data.txt b/source/source-connector/usage-examples/copy-existing-data.txt index dcdac3c..6894320 100644 --- a/source/source-connector/usage-examples/copy-existing-data.txt +++ b/source/source-connector/usage-examples/copy-existing-data.txt @@ -65,7 +65,9 @@ specifying the following configuration options in your source connector: Your source connector copies your collection by creating change event documents that describe inserting each document into your collection. -.. include:: /includes/copy-existing-admonition.rst +.. note:: Data Copy Can Produce Duplicate Events + + .. include:: /includes/copy-existing-admonition.rst To learn more about change event documents, see the :ref:`Change Streams ` guide. diff --git a/source/troubleshooting/recover-from-invalid-resume-token.txt b/source/troubleshooting/recover-from-invalid-resume-token.txt index 807d590..d13f836 100644 --- a/source/troubleshooting/recover-from-invalid-resume-token.txt +++ b/source/troubleshooting/recover-from-invalid-resume-token.txt @@ -161,22 +161,20 @@ property like this: - Records the number of times you reset your connector - Documents to which connector an offset partition belongs - .. example:: + For example, if you named your source connector ``"source-values"`` and you are + setting the ``offset.partition.name`` property for the first time, + configure your connector as follows: - Assume you named your source connector ``"source-values"`` and you are - setting the ``offset.partition.name`` property for the first time. - You would configure your connector as follows: - - .. code-block:: properties + .. code-block:: properties - offset.partition.name=source-values.1 + offset.partition.name=source-values.1 - The next time you reset your connector's offset data, configure - your connector as follows: + The next time you reset your connector's offset data, configure + your connector as follows: - .. code-block:: properties + .. code-block:: properties - offset.partition.name=source-values.2 + offset.partition.name=source-values.2 To learn more about the ``offset.partition.name`` configuration property, see the :ref:`` page.