using spark sftp with databricks

[jch231]

I was wondering if you could help me with using spark-sftp with databricks. Firstly, I am struggling to import the library in databricks - I can only see (very few) examples in the documentation on loading in a dataframe, but nothing on how we import the library into the notebook itself. Secondly, is there a python API for spark-sftp, or is the functionality only available in Scala? (I develop using pyspark by can get past this by loading in the dataframe using scala and creating a temp view to access the dataframe with python). Thanks!

[FurcyPin]

Hello,

All you have to do is make sure the jar of this project is added to your spark environment. I don't use Databricks but I think you could start there: https://docs.databricks.com/libraries.html

As for the pyspark API, it is exactly the same. You can write

df = spark.read.format("com.springml.spark.sftp").options(...).load()

and it should work.

And if you see this error:

java.lang.ClassNotFoundException: com.springml.spark.sftp.DefaultSource

It means that you did not add the jar correctly and that your spark install can't find it.