[SPARK-37313] Child stage using merged output or not should be based on the availability of merged output from parent stage

• Type: Sub-task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.2.1
• Component/s: Shuffle, Spark Core
• Assignee: Unassigned Assign to me
• Reporter: Minchu Yang
• Created: 13/Nov/21 02:48
• Updated: 22/Nov/22 16:44

Description

As discussed in the thread in SPARK-37023, during a stage retry, if parent stage has already generated merged output in the previous attempt, with current behavior, the child stage would not able to fetch the merged output, as this is controlled by dependency.shuffleMergeEnabled (see current implementation here) during the stage retry.

Instead of using a single variable to control behavior at both mapper side (push side) and reducer side (using merged output), whether child stage uses merged output or not must only be based on whether merged output is available for it to use(as discussed here).