[HIVE-13750] Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.1.0
• Component/s: Physical Optimizer
• Target Version/s: 2.1.0
• Assignee: Jesús Camacho Rodríguez
• Reporter: Jesús Camacho Rodríguez
• Created: 12/May/16 18:57
• Updated: 27/Feb/24 22:23
• Resolved: 19/May/16 20:26

Description

Extend ReduceDedup to remove additional shuffle stage created by sorted dynamic partition optimizer when possible, thus avoiding unnecessary work.

By ashutoshc:

Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO) unconditionally adds an extra shuffle stage. If sort columns of previous shuffle and partitioning columns of table match, reduce sink deduplication optimizer removes extra shuffle stage, thus bringing down overhead to zero. However, if they don’t match, we end up doing extra shuffle. This can be improved since we can add table partition columns as a sort columns on earlier shuffle and avoid this extra shuffle. This ensures that in cases query already has a shuffle stage, we are not shuffling data again.

Issue Links

• links to: RB link