Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-48061][SQL][TESTS] Parameterize max limits of `spark.sql.test.…
…randomDataGenerator` ### What changes were proposed in this pull request? This PR aims to parameterize `MAX_ARR_SIZE`, `MAX_MAP_SIZE`, and `MAX_STR_LEN` of `spark.sql.test.randomDataGenerator` by supporting. - `spark.sql.test.randomDataGenerator.maxArraySize` - `spark.sql.test.randomDataGenerator.maxMapSize` - `spark.sql.test.randomDataGenerator.maxStrLen` ### Why are the changes needed? Apache Spark already has the code which needs these parameters. We had better support these to allow the developers to use them without changing and recompiling the source code. https://github.com/apache/spark/blob/0329479acb6758c4d3e53d514ea832a181d31065/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryHashPartitionVerifySuite.scala#L155-L156 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual. **BEFORE (golden file size: `269M`)** ``` $ SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly *StreamingQueryHashPartitionVerifySuite" $ ls -alh ./sql/core/target/scala-2.13/test-classes/structured-streaming/partition-tests/rowsAndPartIds -rw-r--r-- 1 dongjoon staff 269M Apr 30 09:55 ./sql/core/target/scala-2.13/test-classes/structured-streaming/partition-tests/rowsAndPartIds ``` **AFTER (golden file size: `5.8M`)** ``` $ SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly *StreamingQueryHashPartitionVerifySuite" \ -Dspark.sql.test.randomDataGenerator.maxStrLen=100 \ -Dspark.sql.test.randomDataGenerator.maxArraySize=4 $ ls -alh ./sql/core/target/scala-2.13/test-classes/structured-streaming/partition-tests/rowsAndPartIds -rw-r--r-- 1 dongjoon staff 5.8M Apr 30 09:56 ./sql/core/target/scala-2.13/test-classes/structured-streaming/partition-tests/rowsAndPartIds ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46305 from dongjoon-hyun/SPARK-48061. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
- Loading branch information