Why is df.coalesce(1) necessary?

[sunayansaikia]

Hey folks,

Just wanted to understand why 'df.coalesce(1)' was done while writing the dataframe to DFS?
Please refer code here: https://github.com/springml/spark-sftp/blob/master/src/main/scala/com/springml/spark/sftp/DefaultSource.scala#L249

Thanks

[samuel-pt]

@sunayansaikia - That was done to have a single file in SFTP

[sunayansaikia]

Hey @samuel-pt : is this a hard requirement? Can we not download multiple files for upload via SFTP? Can't this option be made configurable?

[samuel-pt]

@sunayansaikia - Its not hard. we can just add a configurable parameter and use it. Following are needed

1. Actual Code changes
2. Tests for the new changes
3. README update accordingly

[sunayansaikia]

ok - cool. Will take a checkout and see.

[shaikmanu797]

@sunayansaikia, the below PR should be able to fix the coalesce numPartitions configuration
#68

[sunayansaikia]

@shaikmanu797 : Great!