[SPARK-19931] InMemoryTableScanExec should rewrite output partitioning and ordering when aliasing output attributes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• Component/s: SQL
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 13/Mar/17 03:40
• Updated: 16/Mar/17 00:19
• Resolved: 16/Mar/17 00:19

Description

Now InMemoryTableScanExec simply takes the outputPartitioning and outputOrdering from the associated InMemoryRelation's child.outputPartitioning and outputOrdering.

However, InMemoryTableScanExec can alias the output attributes. In this case, its outputPartitioning and outputOrdering are not correct and its parent operators can't correctly determine its data distribution.

Issue Links

• is related to: SPARK-19468 Dataset slow because of unnecessary shuffles
• links to: [Github] Pull Request #17175 (viirya)