[HIVE-17035] Optimizer: Lineage transform() should be invoked after rest of the optimizers are invoked

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Logical Optimizer
• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 05/Jul/17 08:42
• Updated: 17/Jul/17 03:53

Description

In a fairly large query which had tens of left join, time taken to create linageInfo itself took 1500+ seconds. This is due to the fact that the table had lots of columns and in some processing, it ended up processing 7000+ value columns in ReduceSinkLineage, though only 50 columns were projected in the query.

It would be good to invoke lineage transform when rest of the optimizers in Optimizer are invoked. This would avoid unwanted processing and help in improving the runtime.

Issue Links

• relates to: HIVE-17036 Lineage: Minor CPU/Mem optimization for lineage transform