[SPARK-22018] Catalyst Optimizer does not preserve top-level metadata while collapsing projects

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.1, 2.2.0
• Fix Version/s: 2.3.0
• Component/s: Optimizer, SQL, Structured Streaming
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 15/Sep/17 00:29
• Updated: 17/May/20 17:58
• Resolved: 15/Sep/17 05:32

Description

If there are two projects like as follows.

Project [a_with_metadata#27 AS b#26]
+- Project [a#0 AS a_with_metadata#27]
   +- LocalRelation <empty>, [a#0, b#1]

Child Project has an output column with a metadata in it, and the parent Project has an alias that implicitly forwards the metadata. So this metadata is visible for higher operators. Upon applying CollapseProject optimizer rule, the metadata is not preserved.

Project [a#0 AS b#26]
+- LocalRelation <empty>, [a#0, b#1]

This is incorrect, as downstream operators that expect certain metadata (e.g. watermark in structured streaming) to identify certain fields will fail to do so.

Issue Links

• links to: [Github] Pull Request #19240 (tdas)