[SPARK-35549] Register merge status even after shuffle dependency is merge finalized

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Shuffle, Spark Core
• Assignee: Unassigned
• Reporter: Venkata krishnan Sowrirajan
• Created: 28/May/21 01:15
• Updated: 09/Jun/21 00:19

Description

Currently the merge statuses which arrive late from the external shuffle services (or shuffle mergers) won't get registered once the shuffle dependency merge is finalized.
This needs to be carefully done as there are lot of corner cases like:
a) executor/node loss causing re-computation due to fetch failure and if the merge statuses gets registered very late then that can cause inconsistencies.
b) similar such cases

cc mridulm80 mshen