[SPARK-22961] Constant columns no longer picked as constraints in 2.3

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Labels: constraints optimizer regression
• Assignee: Adrian Ionescu
• Reporter: Adrian Ionescu
• Created: 04/Jan/18 17:54
• Updated: 31/Jan/18 07:50
• Resolved: 05/Jan/18 13:37

Description

We're no longer picking up x = 2 as a constraint from something like df.withColumn("x", lit(2))

The unit test below succeeds in branch-2.2:

test("constraints should be inferred from aliased literals") {
    val originalLeft = testRelation.subquery('left).as("left")
    val optimizedLeft = testRelation.subquery('left).where(IsNotNull('a) && 'a <=> 2).as("left")

    val right = Project(Seq(Literal(2).as("two")), testRelation.subquery('right)).as("right")
    val condition = Some("left.a".attr === "right.two".attr)

    val original = originalLeft.join(right, Inner, condition)
    val correct = optimizedLeft.join(right, Inner, condition)

    comparePlans(Optimize.execute(original.analyze), correct.analyze)
  }

but fails in branch-2.3 with:

== FAIL: Plans do not match ===
 'Join Inner, (two#0 = a#0)                     'Join Inner, (two#0 = a#0)
!:- Filter isnotnull(a#0)                       :- Filter ((2 <=> a#0) && isnotnull(a#0))
 :  +- LocalRelation <empty>, [a#0, b#0, c#0]   :  +- LocalRelation <empty>, [a#0, b#0, c#0]
 +- Project [2 AS two#0]                        +- Project [2 AS two#0]
    +- LocalRelation <empty>, [a#0, b#0, c#0]      +- LocalRelation <empty>, [a#0, b#0, c#0] 

Issue Links

• links to: [Github] Pull Request #20155 (adrian-ionescu)