[SPARK-38529] Prevent GeneratorNestedColumnAliasing to be applied to non-Explode generators

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.1
• Fix Version/s: 3.3.0
• Component/s: Optimizer
• Assignee: Min Yang
• Reporter: Min Yang
• Created: 12/Mar/22 02:25
• Updated: 12/Dec/22 19:10
• Resolved: 19/May/22 16:52

Description

The Project(_, g: Generate) branch in GeneratorNestedColumnAliasing is only supposed to work for ExplodeBase generators but we do not explicitly return for other types like Inline. Currently the bug is not trigger because there is another bug in the "prune unrequired child" branch in the ColumnPruning which makes other generators like Inline always go to that branch even if it is not applicable.

 

An easy example to show the bug:

Input: <col1: int, col2: array<struct<field1 struct<field1: int, field2: int>, field2 int>>>

Project(field1.field1 as ...)

• Generate(Inline(col2), ..., field1, field2)

 

We will try to incorrectly push the .field1 on field1 into the input of the Inline (col2).

 

 

Issue Links

• links to: [Github] Pull Request #35850 (minyyy)