[SPARK-14854] Left outer join produces incorrect output when the join condition does not have left table key

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.5.1
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: kanika dhuria
• Created: 22/Apr/16 19:05
• Updated: 12/Dec/22 18:10
• Resolved: 29/Jan/17 16:16

Description

import org.apache.spark.sql._
import org.apache.spark.sql.types._

val s = StructType(StructField("num", StringType, true)::Nil)
val s1 = StructType(StructField("num1", StringType, true)::Nil)

val m = sc.textFile("file:/tmp/master.txt").map(_.split(",")).map(p=>Row(p(0)))
val d = sc.textFile("file:/tmp/detail.txt").map(_.split(",")).map(p=>Row(p(0)))
val m1 = sqlContext.createDataFrame(m, s1)
val d1 = sqlContext.createDataFrame(d, s)
val j1 = d1.join(m1,$"num1".===(lit(null)),"left_outer");
j1.take(1)

Returns empty data set. Left table has data.

Issue Links

• relates to: SPARK-17298 Require explicit CROSS join for cartesian products by default