Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Null(Blank) columns are returned with quotes(" ") when writing to csv #78

Open
ucrkarthik opened this issue Aug 31, 2020 · 2 comments
Open

Comments

@ucrkarthik
Copy link

ucrkarthik commented Aug 31, 2020

Would we be able to add a "nullValue" option to the writeToTemp method?

Here is some more info on the nullValue:
nullValue – sets the string representation of a null value. If None is set, it uses the default value, empty string. Since 2.0.1, this nullValue param applies to all supported types including the string type.

https://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html?highlight=drop

Thanks,
Karthik

@ucrkarthik
Copy link
Author

ucrkarthik commented Aug 31, 2020

Here are the changes that I am proposing to add nullValue to the code. This will be in the DefaultSource.scala file.

image

@ucrkarthik
Copy link
Author

@samuel-pt would you be able to review this request?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant