[HIVE-20113] Shuffle avoidance: Disable 1-1 edges for sorted shuffle

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Tez
• Labels: Branch3Candidate
• Assignee: Vineet Garg
• Reporter: Gopal Vijayaraghavan
• Created: 06/Jul/18 22:28
• Updated: 11/Dec/24 15:46
• Resolved: 22/Sep/19 02:13

Description

The sorted shuffle avoidance can have some issues when the shuffle data gets broken up into multiple chunks on disk.

The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to have a final merge at all, it should open a single compressed file and write a single index entry.

Until the shuffle issue is resolved & a lot more testing, it is prudent to disable the optimization for sorted shuffle edges and stop rewriting the RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD).