[TAJO-982] Improve Fetcher to get multiple shuffle outputs through a request

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 0.13.0
• Component/s: Data Shuffle
• Assignee: Jihoon Son
• Reporter: Hyunsik Choi
• Created: 28/Jul/14 05:26
• Updated: 03/Dec/17 23:57

Description

Currently, Fetcher only can request at most a fetch for one shuffle output at a time. The implementation can cause performance degradation even though intermediate data is actually small.

For example, If an input data set of the first stage is big and the intermediate data is very small, QueryMaster will choose a few of nodes for next execution block. Since each node keeps limited threads for fetch, it will take a long time for the nodes for next stage to fetch all intermediate.

If Fetcher can get multiple shuffle outputs through a request, it would solve the slowness which occurs in some cases.