[MAPREDUCE-318] Refactor reduce shuffle code - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.21.0
Component/s: performance, task
Labels:
None

Release Note:

Hide
Refactors shuffle code out of ReduceTask into separate classes in a new package(org.apache.hadoop.mapreduce.task.reduce)
Incorporates ~~MAPREDUCE-240~~, batches up several map output files from a TT to a reducer in a single transfer
Introduces new Shuffle counters to keep track of shuffle errors

Show
Refactors shuffle code out of ReduceTask into separate classes in a new package(org.apache.hadoop.mapreduce.task.reduce) Incorporates MAPREDUCE-240 , batches up several map output files from a TT to a reducer in a single transfer Introduces new Shuffle counters to keep track of shuffle errors

Description

The reduce shuffle code has become very complex and entangled. I think we should move it out of ReduceTask and into a separate package (org.apache.hadoop.mapred.task.reduce). Details to follow.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

HADOOP-5233_api.patch
11/Feb/09 22:33
12 kB
Arun Murthy
HADOOP-5233_part0.patch
12/Feb/09 09:05
21 kB
Arun Murthy
mapred-318-14Aug.patch
14/Aug/09 11:52
233 kB
Jothi Padmanabhan
mapred-318-20Aug.patch
20/Aug/09 16:35
234 kB
Jothi Padmanabhan
mapred-318-24Aug.patch
24/Aug/09 13:02
239 kB
Jothi Padmanabhan
mapred-318-3Sep.patch
03/Sep/09 03:44
242 kB
Jothi Padmanabhan
mapred-318-3Sep-v1.patch
03/Sep/09 06:59
243 kB
Jothi Padmanabhan
mapred-318-common.patch
14/Aug/09 11:52
3 kB
Jothi Padmanabhan

Issue Links

incorporates

MAPREDUCE-268 Implement memory-to-memory merge in the reduce

Resolved

Activity

People

Assignee:: Owen O'Malley

Reporter:: Owen O'Malley

Votes:: 0 Vote for this issue

Watchers:: 33 Start watching this issue

Dates

Created:: 11/Feb/09 18:34

Updated:: 06/Jan/13 06:24

Resolved:: 03/Sep/09 13:56