[MAPREDUCE-1939] split reduce compute phase into two threads one for reading and another for computing

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.20.2
• Component/s: task
• Tags: hadoop task reduce
• Assignee: Unassigned
• Reporter: wangxiaowei
• Created: 14/Jul/10 02:59
• Updated: 10/Mar/15 02:53
• Resolved: 10/Mar/15 02:53

Description

it is known that reduce task is made up of three phases: shuffle , sort and reduce. During reduce phase, a reduce function will read a record from disk or memory first and process it to write to hdfs finally. To convert this serial progress to parallel progress , I split the reduce phase into two threads called producer and consumer individually. producer is used to read record from disk and consumer to process the records read by the first one. I use two buffer, if producer is writing one buffer consumer will read from another buffer. Theoretically there will be a overlap between this two phases so we can reduce the whole reduce time.

I wonder why hadoop does not implement it originally? Is there some potential problems for such ideas ?

I have already implemmented a prototypy. The producer just reads bytes from the disk and leaves the work of transformation to real key and value objects to consumer. The results is not good only a improvement of 13% for time. I think it has someting with the buffer size and the time spending on different threads.Maybe the tiem spend by consumer thread is too long and the producer has to wait until the next buffer is available.