Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add default max batch size and batchSize parameter for SF updates #55

Open
wants to merge 41 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
46e4f57
add batchSize parameter for controlling bulk upload batch size. defau…
unintellisense Jun 28, 2020
144f9f9
update readme
unintellisense Jun 28, 2020
e3f58b0
Merge pull request #1 from loanpal-engineering/add-batch-size-update
unintellisense Jun 28, 2020
bd21e6a
use updated wave-api dependency
unintellisense Jul 12, 2020
707921d
null instead of empty for null objects
unintellisense Jul 14, 2020
af32243
cast null as null
unintellisense Jul 14, 2020
6b4fb6d
use master
unintellisense Jul 14, 2020
eda9081
Merge pull request #2 from loanpal-engineering/nulls-not-empty
unintellisense Jul 14, 2020
131c4f3
spark 3.0/scala 2.12 compatibility
SKinserLoanpal Oct 1, 2020
67ec60a
change to jitpack
SKinserLoanpal Oct 8, 2020
b8c10ea
Merge pull request #3 from loanpal-engineering/spark-3.0
SKinserLoanpal Oct 8, 2020
32effa7
swap out spark version
SKinserLoanpal Oct 8, 2020
0e62d2a
Merge pull request #4 from loanpal-engineering/spark-3.0
SKinserLoanpal Oct 8, 2020
8f72643
use salesforce magic null string value
SKinserLoanpal Feb 1, 2021
e9b423c
maybe empty string is dumb
SKinserLoanpal Feb 2, 2021
d934530
make note of reading/writing
SKinserLoanpal Feb 2, 2021
dedec34
switch to cast
SKinserLoanpal Feb 2, 2021
5f4437f
fix
SKinserLoanpal Feb 2, 2021
6770197
a
SKinserLoanpal Feb 2, 2021
c1d8e0b
add empty string na
SKinserLoanpal Feb 2, 2021
2451cb6
make all null values #N/A aside from dates
SKinserLoanpal Feb 2, 2021
7e068e5
fix dumb mistake
SKinserLoanpal Feb 2, 2021
03491b1
add special case for booleans
SKinserLoanpal Feb 2, 2021
3f9358d
dumb mistake
SKinserLoanpal Feb 2, 2021
f7ed7f2
fix data writer
SKinserLoanpal Feb 2, 2021
99c125d
fix
SKinserLoanpal Feb 2, 2021
4661c15
delete commented code
SKinserLoanpal Feb 3, 2021
e226b20
Merge pull request #5 from loanpal-engineering/null-na
SKinserLoanpal Feb 4, 2021
b6fe136
add support for max column width to csv parser (#6)
unintellisense Apr 27, 2022
7581bf8
shade dependency
SKinserLoanpal Nov 15, 2022
6706341
Merge pull request #8 from loanpal-engineering/spark-3.3.0
SKinserLoanpal Nov 15, 2022
b19abd0
support pkChunks with filtering (handling empty batches)
mweldon-loanpal Mar 17, 2023
fab2d96
Merge pull request #9 from loanpal-engineering/BI-10792
unintellisense Mar 17, 2023
6a658f1
Create codeowners
gwadley-goodleap May 24, 2023
850f035
bulk api 2.0
SKinserLoanpal Nov 29, 2023
e1d7e59
fix
SKinserLoanpal Jan 11, 2024
9a04a88
Merge pull request #10 from loanpal-engineering/spark-3.3.0
SKinserLoanpal Jan 11, 2024
7316f09
change force api to 53
SKinserLoanpal Jan 16, 2024
2d727e8
Merge pull request #11 from loanpal-engineering/spark-3.3.0
SKinserLoanpal Jan 16, 2024
8ccd432
fix max columns
SKinserLoanpal May 23, 2024
bd9c2fa
Merge pull request #14 from loanpal-engineering/spark-3.3.0
SKinserLoanpal May 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
support pkChunks with filtering (handling empty batches)
mweldon-loanpal committed Mar 17, 2023
commit b19abd06f548ea0df532a61f786cefd393f91427
8 changes: 4 additions & 4 deletions pom.xml
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@
<artifactId>spark-salesforce</artifactId>
<packaging>jar</packaging>
<description>spark-salesforce</description>
<version>1.1.4</version>
<version>1.1.5</version>
<name>spark-salesforce</name>
<organization>
<name>com.springml</name>
@@ -68,9 +68,9 @@
<version>40.0.0</version>
</dependency>
<dependency>
<groupId>com.springml</groupId>
<groupId>com.github.loanpal-engineering</groupId>
<artifactId>salesforce-wave-api</artifactId>
<version>1.0.8-loanpal</version>
<version>v1.1.0</version>
</dependency>
<!-- <dependency>-->
<!-- <groupId>com.github.loanpal-engineering</groupId>-->
@@ -97,7 +97,7 @@
<dependency>
<groupId>com.github.loanpal-engineering</groupId>
<artifactId>java-sizeof</artifactId>
<version>6d786c8</version>
<version>0.1</version>
</dependency>
<dependency>
<groupId>org.codehaus.woodstox</groupId>
Original file line number Diff line number Diff line change
@@ -72,6 +72,7 @@ case class BulkRelation(

// Use Csv parser to split CSV by rows to cover edge cases (ex. escaped characters, new line within string, etc)
def splitCsvByRows(csvString: String): Seq[String] = {
if (csvString == "Records not found for this query") Seq.empty
// The CsvParser interface only interacts with IO, so StringReader and StringWriter
val inputReader = new StringReader(csvString)