Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22961

Constant columns no longer picked as constraints in 2.3

    Details

      Description

      We're no longer picking up x = 2 as a constraint from something like df.withColumn("x", lit(2))

      The unit test below succeeds in branch-2.2:

      test("constraints should be inferred from aliased literals") {
          val originalLeft = testRelation.subquery('left).as("left")
          val optimizedLeft = testRelation.subquery('left).where(IsNotNull('a) && 'a <=> 2).as("left")
      
          val right = Project(Seq(Literal(2).as("two")), testRelation.subquery('right)).as("right")
          val condition = Some("left.a".attr === "right.two".attr)
      
          val original = originalLeft.join(right, Inner, condition)
          val correct = optimizedLeft.join(right, Inner, condition)
      
          comparePlans(Optimize.execute(original.analyze), correct.analyze)
        }
      

      but fails in branch-2.3 with:

      == FAIL: Plans do not match ===
       'Join Inner, (two#0 = a#0)                     'Join Inner, (two#0 = a#0)
      !:- Filter isnotnull(a#0)                       :- Filter ((2 <=> a#0) && isnotnull(a#0))
       :  +- LocalRelation <empty>, [a#0, b#0, c#0]   :  +- LocalRelation <empty>, [a#0, b#0, c#0]
       +- Project [2 AS two#0]                        +- Project [2 AS two#0]
          +- LocalRelation <empty>, [a#0, b#0, c#0]      +- LocalRelation <empty>, [a#0, b#0, c#0] 
      

        Attachments

          Activity

            People

            • Assignee:
              a.ionescu Adrian Ionescu
              Reporter:
              a.ionescu Adrian Ionescu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: