[SPARK-24237] continuous shuffle dependency

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.4.0
• Component/s: Structured Streaming
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Jose Torres
• Created: 10/May/18 01:38
• Updated: 25/May/21 01:55
• Resolved: 25/May/21 01:45

Description

https://docs.google.com/document/d/1IL4kJoKrZWeyIhklKUJqsW-yEN7V7aL05MmM65AYOfE/edit#heading=h.8t3ci57f7uii

 

We might not need this to be an actual org.apache.spark.Dependency. We need to somehow register with MapOutputTracker, or write our own custom tracker if this ends up being infeasible.

Issue Links

• links to: [Github] Pull Request #21293 (xuanyuanking)
• links to: GitHub Pull Request #21293