pyspark-session-2021-10-25-RDD-join.txt

Inner Join Example

$ pyspark
Python 3.7.10 (default, Jun  3 2021, 00:02:01)
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.4.5-amzn-0
      /_/

Using Python version 3.7.10 (default, Jun  3 2021 00:02:01)
SparkContext available as 'sc'.
SparkSession available as 'spark'.
>>>
>>>
>>> x = sc.parallelize([("a", 1), ("a", 4), ("b", 7), ("b", 8), ("c", 89)])
>>> y = sc.parallelize([("a", 100), ("a", 400), ("b", 700), ("b", 800), ("b", 900), ("d", 890)])
>>> x.collect()
[
 ('a', 1), ('a', 4), 
 ('b', 7), ('b', 8), 
 ('c', 89)
]
>>> y.collect()
[
 ('a', 100), ('a', 400), 
 ('b', 700), ('b', 800), ('b', 900), 
 ('d', 890)
]

>>> joined = x.join(y)
>>> joined.collect()
[
 ('b', (7, 800)), 
 ('b', (7, 900)), 
 ('b', (7, 700)), 
 ('b', (8, 800)), 
 ('b', (8, 900)), 
 ('b', (8, 700)), 
 ('a', (1, 100)), 
 ('a', (1, 400)), 
 ('a', (4, 100)), 
 ('a', (4, 400))
]
>>> joined2 = y.join(x)
>>> joined2.collect()
[
 ('b', (700, 8)), 
 ('b', (700, 7)), 
 ('b', (800, 8)), 
 ('b', (800, 7)), 
 ('b', (900, 8)), 
 ('b', (900, 7)), 
 ('a', (100, 4)), 
 ('a', (100, 1)), 
 ('a', (400, 4)), 
 ('a', (400, 1))
]
>>>