Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
Description
The sorted shuffle avoidance can have some issues when the shuffle data gets broken up into multiple chunks on disk.
The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to have a final merge at all, it should open a single compressed file and write a single index entry.
Until the shuffle issue is resolved & a lot more testing, it is prudent to disable the optimization for sorted shuffle edges and stop rewriting the RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD).