-
-
Notifications
You must be signed in to change notification settings - Fork 355
SharedSparkContext
Holden Karau edited this page Aug 22, 2017
·
4 revisions
Instead of initializing SparkContext
before every test case or per class you can easily get your SparkContext
by extending SharedSparkContext
. SharedSparkContext
initializes SparkContext
before all test cases and stops this context after all test cases. For Spark 2.2 and higher you can also share the SparkContext
(and SparkSession
if in DataFrame
tests) between tests by adding override implicit def reuseContextIfPossible: Boolean = true
to your test.
For Java users the same functionality is supported by SharedJavaSparkContext
.
Example:
class SampleTest extends FunSuite with SharedSparkContext {
test("test initializing spark context") {
val list = List(1, 2, 3, 4)
val rdd = sc.parallelize(list)
assert(rdd.count === list.length)
}
}