[MAPREDUCE-5786] Support Keep-Alive connections in ShuffleHandler

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Labels: shuffle
• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 09/Mar/14 00:19
• Updated: 21/Mar/14 21:46
• Resolved: 21/Mar/14 21:46

Description

Currently ShuffleHandler supports fetching map-outputs in batches from same host. But there are scenarios wherein, fetchers pull data aggressively (i.e start pulling the data as & when they are available). In this case, the number of mapIds that are pulled from same host remains at 1. This causes lots of connections to be established.

Number of connections can be reduced a lot if ShuffleHandler supports Keep-Alive.

Sub-Tasks

• Modify ShuffleHandler to support Keep-Alive: Closed
• Modify Fetcher to pull data using persistent connection: Resolved