[FLINK-13244] Delayed Scheduler in Flink

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Component/s: flink-contrib
• Labels: auto-deprioritized-major auto-deprioritized-minor
• Assignee: Unassigned
• Reporter: Mridul Verma
• Created: 12/Jul/19 10:05
• Updated: 24/Nov/21 22:38

Description

Currently with the flink scheduler and local splits

• Suppose i use LocalSplitter to create splits for per hostname
• Now a host requests for a local splits, there are chances that this node will get a local split but in case of no local split present , this node might get remote split and hence the data locality is of no use in these cases because there might be just some other node just round the corner who was just going to ask for its own local split but because this node asked for the split first , both the hosts/node will be given remote splits and hence the overall throughput of the system might decrease.
• Proposal is to use Delayed scheduling. This has shown to be quite effective against these cases and might help us increase the overall throughput given the latency difference between execution of local and remote split is significant.
• https://cs.stanford.edu/~matei/papers/2010/eurosys_delay_scheduling.pdf
  *