[HADOOP-3366] Shuffle/Merge improvements - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.18.0
Component/s: None
Labels:
None

Release Note:
Improved shuffle so that all fetched map-outputs are kept in-memory before being merged by stalling the shuffle so that the in-memory merge executes and frees up memory for the shuffle.

Description

This is intended to be a meta-issue to track various improvements to shuffle/merge in the reducer.

Attachments

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

3366.1.patch
16/May/08 13:09
9 kB
Devaraj Das
3366.1.patch
16/May/08 13:15
9 kB
Devaraj Das
ifile.patch
19/May/08 11:06
9 kB
Arun Murthy
HADOOP-3366_0_20080605.patch
06/Jun/08 06:59
10 kB
Arun Murthy
3366.reducetask.patch
06/Jun/08 14:27
3 kB
Devaraj Das
HADOOP-3366_2_20080606.patch
06/Jun/08 23:20
15 kB
Arun Murthy
HADOOP-3366_2_20080606.patch
06/Jun/08 23:48
16 kB
Arun Murthy

Issue Links

depends upon

HADOOP-2095 Reducer failed due to Out ofMemory

Closed

HADOOP-3365 SequenceFile.Sorter.MergeQueue.next does an unnecessary copy of the key

Closed

HADOOP-3297 The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved

Closed

Activity

People

Assignee:: Arun Murthy

Reporter:: Arun Murthy

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 08/May/08 07:14

Updated:: 08/Jul/09 16:52

Resolved:: 07/Jun/08 00:09