[#HADOOP-3366] Shuffle/Merge improvements

Hadoop Common

Created: 08/May/08 07:14 AM Updated: 08/Jul/09 04:52 PM

Return to search

Component/s:

None

Affects Version/s:

None

Fix Version/s:

0.18.0

Time Tracking:

Not Specified

File Attachments:

Issue Links:

dependent

This issue depends upon:

~~HADOOP-3365~~ SequenceFile.Sorter.MergeQueue.next does an unnecessary copy of the key

~~HADOOP-2095~~ Reducer failed due to Out ofMemory

~~HADOOP-3297~~ The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved

Release Note:	Improved shuffle so that all fetched map-outputs are kept in-memory before being merged by stalling the shuffle so that the in-memory merge executes and frees up memory for the shuffle.
Resolution Date:	07/Jun/08 12:09 AM

Description

« Hide

This is intended to be a meta-issue to track various improvements to shuffle/merge in the reducer.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

Repository	Revision	Date	User	Message
ASF	#664208	Sat Jun 07 00:08:54 UTC 2008	acmurthy	~~HADOOP-3366~~. Stall the shuffle while in-memory merge is in progress.
				Files Changed
				MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java MODIFY /hadoop/core/trunk/CHANGES.txt MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/mapred/MRConstants.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/mapred/RamManager.java