[SPARK-31801] Register shuffle map output metadata with a shuffle output tracker

• Type: Sub-task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Shuffle
• Assignee: Unassigned
• Reporter: Matt Cheah
• Created: 23/May/20 01:38
• Updated: 14/Dec/20 13:30

Description

Part of the design as discussed in this document.

Establish a ShuffleOutputTracker API that resides on the driver, and handle accepting map output metadata returned by the map output writers and send them to the output tracker module accordingly.

Requires https://issues.apache.org/jira/browse/SPARK-31798.

Issue Links

• links to: [Github] Pull Request #28618 (mccheah)
• links to: [Github] Pull Request #28902 (mccheah)
• links to: [Github] Pull Request #30763 (attilapiros)