Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOCSP-31213 - streaming config #183

Merged
merged 67 commits into from
Nov 14, 2023
Merged
Show file tree
Hide file tree
Changes from 60 commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
5477768
first test
mongoKart Oct 9, 2023
83ebbd1
fix
mongoKart Oct 9, 2023
79cd158
fix
mongoKart Oct 9, 2023
7e1df0e
test
mongoKart Oct 9, 2023
b20a31a
add streaming
mongoKart Oct 9, 2023
d7292d7
cache config
mongoKart Oct 10, 2023
4a4f581
remove configexception
mongoKart Oct 10, 2023
2c921ba
add streaming to index
mongoKart Oct 10, 2023
7332e54
streaming mode
mongoKart Oct 10, 2023
4b9aee2
fix
mongoKart Oct 13, 2023
76509b3
fix
mongoKart Oct 13, 2023
716b687
fix
mongoKart Oct 13, 2023
8ee6d4c
-a
mongoKart Oct 13, 2023
6e9504c
-a
mongoKart Oct 13, 2023
b92f2ae
wip
mongoKart Oct 13, 2023
608f784
wip
mongoKart Oct 16, 2023
832ebd5
wip
mongoKart Oct 16, 2023
dbf7f33
batch config
mongoKart Oct 16, 2023
c5a8102
wip
mongoKart Oct 16, 2023
ee5a600
connection config
mongoKart Oct 17, 2023
bbec2b9
shared config
mongoKart Oct 17, 2023
f31a54d
wip
mongoKart Oct 17, 2023
94692b3
wip
mongoKart Oct 17, 2023
f2343d9
wip
mongoKart Oct 17, 2023
42fc169
test
mongoKart Oct 17, 2023
7dbc14a
batch read config
mongoKart Oct 17, 2023
bd292c5
stream read config
mongoKart Oct 18, 2023
8197a85
stream write
mongoKart Oct 18, 2023
4ca9053
streaming mode
mongoKart Oct 18, 2023
c056fd3
java batch writes
mongoKart Oct 19, 2023
8a9d61d
batch write languages
mongoKart Oct 19, 2023
164565a
factor out table
mongoKart Oct 19, 2023
80cedb2
wip
mongoKart Oct 19, 2023
aba5ac7
fixes
mongoKart Oct 19, 2023
bb03332
fix
mongoKart Oct 19, 2023
1c6f1df
api links
mongoKart Oct 19, 2023
b733084
Merge branch 'master' of ssh://github.com/mongodb/docs-spark-connecto…
mongoKart Oct 19, 2023
113c446
fixes
mongoKart Oct 19, 2023
47f0752
java batch read
mongoKart Oct 20, 2023
4ee230a
java python reads
mongoKart Oct 20, 2023
f1b5bc8
scala read
mongoKart Oct 20, 2023
3d3704d
api links
mongoKart Oct 20, 2023
7664a86
remove driver dropdown
mongoKart Oct 20, 2023
41892a1
refactor inference intro
mongoKart Oct 20, 2023
2a6c88a
remove inline scala link
mongoKart Oct 20, 2023
4304c14
batch write changes
mongoKart Oct 20, 2023
45ad35e
stream read api docs
mongoKart Oct 20, 2023
9d1178f
streaming write
mongoKart Oct 20, 2023
609942e
remove write settings
mongoKart Oct 20, 2023
ef836d0
fix link text
mongoKart Oct 20, 2023
e33546b
fixes
mongoKart Oct 20, 2023
6d4a3b3
configexception
mongoKart Oct 20, 2023
392f8ce
vale fixes
mongoKart Oct 20, 2023
776ce65
vale/style guide
mongoKart Oct 20, 2023
ebc29c8
pa review
mongoKart Oct 30, 2023
5452d7b
pa feedback
mongoKart Oct 31, 2023
0bce65d
tab test
mongoKart Oct 31, 2023
0260735
move inference and filters
mongoKart Oct 31, 2023
ad08c26
resize heading
mongoKart Oct 31, 2023
c0a08da
redirects
mongoKart Oct 31, 2023
ad4d716
js feedback
mongoKart Nov 13, 2023
5531cae
js feedback
mongoKart Nov 14, 2023
a70c300
js feedback
mongoKart Nov 14, 2023
5be820d
js feedback
mongoKart Nov 14, 2023
949a40b
js feedback
mongoKart Nov 14, 2023
68c7424
fixes
mongoKart Nov 14, 2023
719161b
Merge remote-tracking branch 'upstream/master' into docsp-31213-strea…
mongoKart Nov 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion config/redirects
Original file line number Diff line number Diff line change
Expand Up @@ -61,4 +61,9 @@ raw: ${prefix}/sparkR -> ${base}/v3.0/r-api/
[*-v3.0]: ${prefix}/${version}/configuration/read -> ${base}/${version}/
[*-v3.0]: ${prefix}/${version}/write-to-mongodb -> ${base}/${version}/
[*-v3.0]: ${prefix}/${version}/read-from-mongodb -> ${base}/${version}/
[*-v3.0]: ${prefix}/${version}/structured-streaming -> ${base}/${version}/
[*-v3.0]: ${prefix}/${version}/structured-streaming -> ${base}/${version}/
[v10.0-*]: ${prefix}/${version}/configuration/write -> ${base}/${version}/batch-mode/batch-write-config/
[v10.0-*]: ${prefix}/${version}/configuration/read -> ${base}/${version}/batch-mode/batch-read-config/
[v10.0-*]: ${prefix}/${version}/write-to-mongodb -> ${base}/${version}/batch-mode/batch-write/
[v10.0-*]: ${prefix}/${version}/read-from-mongodb -> ${base}/${version}/batch-mode/batch-read/
[v10.0-*]: ${prefix}/${version}/structured-streaming -> ${base}/${version}/streaming-mode/
10 changes: 9 additions & 1 deletion snooty.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,15 @@ title = "MongoDB Spark Connector"

intersphinx = ["https://www.mongodb.com/docs/manual/objects.inv"]

toc_landing_pages = ["configuration"]
toc_landing_pages = [
"configuration",
"/batch-mode",
"/streaming-mode",
"/streaming-mode/streaming-read",
"/streaming-mode/streaming-write",
"/batch-mode/batch-write",
"/batch-mode/batch-read",
]

[constants]
connector-short = "Spark Connector"
Expand Down
32 changes: 32 additions & 0 deletions source/batch-mode.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
==========
Batch Mode
==========

.. contents:: On this page
:local:
:backlinks: none
:depth: 1
:class: singlecol

.. toctree::

/batch-mode/batch-read
/batch-mode/batch-write

Overview
--------

In batch mode, you can use the Spark Dataset and DataFrame APIs to process data at
a specified time interval.

The following sections show you how to use the {+connector-short+} to read data from
MongoDB and write data to MongoDB in batch mode:

- :ref:`batch-read-from-mongodb`
- :ref:`batch-write-to-mongodb`

.. tip:: Apache Spark Documentation

To learn more about using Spark to process batches of data, see the
`Spark Programming Guide
<https://spark.apache.org/docs/latest/sql-programming-guide.html>`__.
Loading
Loading