Skip to content

Commit

Permalink
connection config
Browse files Browse the repository at this point in the history
  • Loading branch information
mongoKart committed Oct 17, 2023
1 parent c5a8102 commit ee5a600
Show file tree
Hide file tree
Showing 4 changed files with 41 additions and 72 deletions.
35 changes: 1 addition & 34 deletions source/batch-mode/batch-write-config.txt
Original file line number Diff line number Diff line change
Expand Up @@ -169,37 +169,4 @@ You can configure the following properties when writing data to MongoDB in batch
guide on the
:manual:`WriteConcern wtimeout option </reference/write-concern/#wtimeout>`.

.. _configure-batch-output-uri:

``connection.uri`` Configuration Setting
----------------------------------------

You can set all :ref:`spark-batch-output-conf` via the write ``connection.uri``.

.. note::

If you use ``SparkConf`` to set the connector's write configurations,
prefix ``spark.mongodb.write.`` to the setting.

.. code:: cfg

spark.mongodb.write.connection.uri=mongodb://127.0.0.1/test.myCollection

The configuration corresponds to the following separate configuration
settings:

.. code:: cfg

spark.mongodb.write.connection.uri=mongodb://127.0.0.1/
spark.mongodb.write.database=test
spark.mongodb.write.collection=myCollection

If you specify a setting both in the ``connection.uri`` and in a separate
configuration, the ``connection.uri`` setting overrides the separate
setting. For example, in the following configuration, the
database for the connection is ``foobar``:

.. code:: cfg

spark.mongodb.write.connection.uri=mongodb://127.0.0.1/foobar
spark.mongodb.write.database=bar
.. include:: /includes/connection-write-config.rst
2 changes: 1 addition & 1 deletion source/configuration.txt
Original file line number Diff line number Diff line change
Expand Up @@ -123,4 +123,4 @@ available. You must specify these settings by using a JVM system property.

.. tip:: System Properties

For more information on Java system properties, see the `Java documentation. <https://docs.oracle.com/javase/tutorial/essential/environment/sysprop.html>`__
For more information on Java system properties, see the `Java documentation. <https://docs.oracle.com/javase/tutorial/essential/environment/sysprop.html>`__
34 changes: 34 additions & 0 deletions source/includes/connection-write-config.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
Specifying Properties in ``connection.uri``
-------------------------------------------

If you're using ``SparkConf``, you can specify all the previous settings either
individually or combined in the write ``connection.uri`` setting.

The following code examples show how to specify ``connection.uri``,
``database``, and ``collection`` settings both separately and in ``connection.uri``:

- Separately:

.. code:: cfg
spark.mongodb.write.connection.uri=mongodb://127.0.0.1/
spark.mongodb.write.database=test
spark.mongodb.write.collection=myCollection
- In ``connection.uri``:

.. code:: cfg
spark.mongodb.write.connection.uri=mongodb://127.0.0.1/test.myCollection
.. important:: connection.uri Takes Precedence

If you specify a setting in both the ``connection.uri`` and on its own line,
the ``connection.uri`` setting takes precedence.
For example, in the following configuration, the connection
database is ``foobar``:

.. code:: cfg
spark.mongodb.write.connection.uri=mongodb://127.0.0.1/foobar
spark.mongodb.write.database=bar
42 changes: 5 additions & 37 deletions source/streaming-mode/streaming-write-config.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,12 @@ Write Configuration Options
:class: singlecol

.. _spark-streaming-output-conf:
.. _spark-output-conf:

Write Configuration
-------------------

You can configure the following properties when writing data to MongoDB in streaming mode:
You can configure the following properties when writing data to MongoDB in streaming mode.

.. note::

Expand Down Expand Up @@ -65,44 +66,11 @@ You can configure the following properties when writing data to MongoDB in strea
| For more information about checkpoints, see the
`Spark Structured Streaming Programming Guide <https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing>`__
|
| **Default:**
| **Default:** None

* - ``forceDeleteTempCheckpointLocation``
- | A Boolean value that specifies whether to delete existing checkpoint data.
|
| **Default:**

.. _configure-streaming-output-uri:

``connection.uri`` Configuration Setting
----------------------------------------

You can set all :ref:`spark-output-conf` via the write ``connection.uri``.

.. note::

If you use ``SparkConf`` to set the connector's write configurations,
prefix ``spark.mongodb.write.`` to the setting.

.. code:: cfg

spark.mongodb.write.connection.uri=mongodb://127.0.0.1/test.myCollection

The configuration corresponds to the following separate configuration
settings:

.. code:: cfg

spark.mongodb.write.connection.uri=mongodb://127.0.0.1/
spark.mongodb.write.database=test
spark.mongodb.write.collection=myCollection

If you specify a setting both in the ``connection.uri`` and in a separate
configuration, the ``connection.uri`` setting overrides the separate
setting. For example, in the following configuration, the
database for the connection is ``foobar``:

.. code:: cfg
| **Default:** False

spark.mongodb.write.connection.uri=mongodb://127.0.0.1/foobar
spark.mongodb.write.database=bar
.. include:: /includes/connection-write-config.rst

0 comments on commit ee5a600

Please sign in to comment.