|
An interesting observation regarding using the ramfs. I guess i should raise a separate jira but let me put it here anyway -
I had a job (loadgen from hadoop-test) consisting of 2500 maps and 1 reducer. The ramfs size was 300MB and io.sort.factor was 100. The cluster had 20 nodes. Each map generated 5 MB of data. The amount of time it took to complete the job was 45 minutes (with the above changes). The number of files that missed the ramfs and ended up on disk was ~2000. I ran the same job (with exactly the same config) with the reducer throttled - if a ramfs merge is on, it would wait for that to complete before fetching anything new. This basically results in all files ending up in the ramfs. The job ran in 30 minutes. So although I didn't notice any significant performance gain for this job with the shuffle protocol changes as proposed in my last comment but in general it looks like this is going to be true - for a given job, if we have a faster shuffle, more files get created on the disk, and depending on the number/size of map outputs for the job, this might adversely affects the final merge, thereby affecting the overall runtime of the job. I will see if the above behavior can be modelled. Under what condition fetched map outputs will end up on disk directly? This is related to hadoop-2095. They should be considered together. I ran a benchmark (loadgen) with the attached patch. Here are the details:
1) Num maps - 10000 2) Size of each map output - 1KB 3) Size of cluster - 80 nodes 4) Num reducers - 1 With the patch, the run took ~7 minutes. On trunk, the same job took ~11 minutes. How long did the map phase take?
The map phase took roughly 3 minutes
this patch does not include waiting on the reducer fetch if the memory fs is full ? or does it ?
No it doesn't. That should be outside the scope of this one.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381308/3297.patch  against trunk revision 653638. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2406/testReport/ This message is automatically generated. Fixed findbugs warnings. The test failure is not related to the patch.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381496/3297.patch against trunk revision 653749. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2412/testReport/ This message is automatically generated. the patch looks good... the only concern I have is that if we want to check and see that it does not degrade perofrmace and lead to more problems at the jobtracker or otherwise.
We could try running sort with this patch or some map reduce job where we have a huge number of mappers say 100,000 and 500 reducers or something. This is just to verify that the task tracker being agressive in fetching the mapoutputs does not degrade performance. The maps are short lived, so its not hard to imagine a situation that all the reduces start bombarding the jobtracker with requests for maps at the same time asking for more. We should check to see if the jobtracker can handle the load and the performance in such a situation does not degrade. Yes, I ran large jobs with 250000 maps and 400 reducers on 250 nodes and saw no performance issues.
thats great...
+1 for commit. I just committed this. Thanks, Devaraj!
Integrated in Hadoop-trunk #484 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/484/
) This patch should be in release 17. This jira was created because of hadoop-3327
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Improvement
Closed
Major
The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
1) The TaskTracker polls the JobTracker asking for 500 task completion events. If it gets the full payload, it immediately asks for another bunch of 500 and so on. When it gets less than 500, it switches to current behavior - sleep for a fixed amount of time (heartbeat interval). A small number of events per RPC would ensure that each RPC takes a lesser amount of time although the number of RPCs would be more.
2) The Task asks for 10000 events at a time every second from the TaskTracker.