avoid shuffle?

[tooptoop4]

i read csv input then write parquet with partitionby, it takes a long time. any settings u recommend ? maybe https://issues.apache.org/jira/browse/SPARK-24940 ?

[lyogev]

So if you're reading a single CSV and perform partitionby the first part will be a single partition.
You can either repartition using the hints in the link or create a multi config job with a CSV repartitioned to parquet (using the repartition flag in the parquet output) and then reading and running your business logic on top of it