Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33035

Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.1, 3.1.0
    • Fix Version/s: 3.0.2, 3.1.0
    • Component/s: SQL
    • Labels:
      None

      Description

      This ticket aims at fixing corner-case bugs in the `QueryPlan#transformUpWithNewOutput` that is used to propagate updated `ExprId`s in a bottom-up way. Let's say we have a rule to simply assign new `ExprId`s in a projection list like this;

      case class TestRule extends Rule[LogicalPlan] {
        override def apply(plan: LogicalPlan): LogicalPlan = plan.transformUpWithNewOutput {
          case p @ Project(projList, _) =>
            val newPlan = p.copy(projectList = projList.map { _.transform {
              // Assigns a new `ExprId` for references
              case a: AttributeReference => Alias(a, a.name)()
            }}.asInstanceOf[Seq[NamedExpression]])
      
            val attrMapping = p.output.zip(newPlan.output)
            newPlan -> attrMapping
        }
      }
      

      Then, this rule is applied into a plan below;

      (3) Project [a#5, b#6]
      +- (2) Project [a#5, b#6]
         +- (1) Project [a#5, b#6]
            +- LocalRelation <empty>, [a#5, b#6]
      

      In the first transformation, the rule assigns new `ExprId`s in `(1) Project` (e.g., a#5 AS a#7, b#6 AS b#8). In the second transformation, the rule corrects the input references of `(2) Project` first by using attribute mapping given from `(1) Project` (a#5->a#7 and b#6->b#8) and then assigns new `ExprId`s (e.g., a#7 AS a#9, b#8 AS b#10). But, in the third transformation, the rule fails because it tries to correct the references of `(3) Project` by using incorrect attribute mapping (a#7->a#9 and b#8->b#10) even though the correct one is a#5->a#9 and b#6->b#10.

        Attachments

          Activity

            People

            • Assignee:
              maropu Takeshi Yamamuro
              Reporter:
              maropu Takeshi Yamamuro
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: