Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22141

Propagate empty relation before checking Cartesian products

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.2.0
    • 2.2.1, 2.3.0
    • SQL
    • None

    Description

      When inferring constraints from children, Join's condition can be simplified as None.
      For example,
      val testRelation = LocalRelation('a.int)
      val x = testRelation.as("x")
      val y = testRelation.where($"a" === 2 && !($"a" === 2)).as("y")
      x.join.where($"x.a" === $"y.a")

      The plan will become

      Join Inner
      :- LocalRelation <empty>, a#23
      +- LocalRelation <empty>, a#224

      And the Cartesian products check will throw exception.
      Propagate empty relation before checking Cartesian products, and the issue is resolved.

      Attachments

        Activity

          People

            Gengliang.Wang Gengliang Wang
            Gengliang.Wang Gengliang Wang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: