|
[
Permlink
| « Hide
]
Suhas Gogate added a comment - 15/Sep/08 11:59 PM
patch added
This JIRA takes care of post execution diagnosis of map/reduce job. The linked jira (HADOOP-3956) would extend it further for runtime progress analysis of Map/Reduce jobs.
This would be very useful to have. Can jobdetails.jsp or jobtracker.jsp (may be via StatusHttpServlet) serve this analysis after the job is done? That will help users to check how job performed directly from webUI.
PS : Definition of Vaidya  Lohit,
This is the first version of Hadoop Vaidya, which works only in the postmortem analysis mode. Later, as needed, it can be extended to do analysis of a running job too. P.S. Can you review the patch ? Thanks. Comments on the coding style - Indentation needs to be fixed to conform to hadoop's coding conventions. Also the variable names are better read without the underscore prefix.
moving this from 0.19 due to feature freeze.
Thanks Lohit, Mac & Devraj for providing the comments. Attaching the new patch incorporating comments to make it comply with hadoop coding conventions.
Thanks. Hi Suhas, Lohith
We are students from University of Minnesota looking for a project for our distributed systems class. We found this project to be quite interesting. We would like to work on the extension for run time analysis. How feasible is that considering we have a time frame of 4 weeks to complete our project. Thanks Sachin Enis, I believe the feature freeze applies to hadoop core, and not contrib, right ?
Hi Sachin, I sent you a detailed reply to your email couple of days back. Yes there is a possibility, although need to know few details about your familiarity with hadoop M/R and what is the start date for your project etc. We can discuss it off-line to keep JIRA updates related to technical discussion. will send you the email again today. thx. --Suhas
Making it patch available.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12392443/hadoop-vaidya-0.2.patch against trunk revision 709040. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 23 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3509/console This message is automatically generated. error applying patch. adding new patch.
patch submitted along with documentation.
latest patch file (hadoop_vaidya.0.3.patch) attached.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12393197/hadoop_vaidya.0.3.patch against trunk revision 709609. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 20 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3517/testReport/ This message is automatically generated. Patch added for branch 0.19.0
This can't go into 0.19, because it is new functionality.
Why is it in chukwa, rather than some where more global? Does it depend on chukwa? 1. Although it is an independent tool right now, eventually would be integrated with Chukwa (part of health care center), providing rule based analysis of the data it collects for M/R jobs. At this time it does not use any Chukwa interface/library etc.
2. As Milind mentioned in one of his earlier comments, I was under impression that feature freeze applies to hadoop core and not "contrib". Is it true? 1. I think it would add value outside of Chukwa and be usable without Chukwa. Maybe the right place is another contrib project?
2. No. 0.19 is closed for both core and contrib. Owen, Sorry, I did not mean that Hadoop Vaidya would loose its independent identity after integration with Chukwa. Optionally It should also run outside Chukwa framework, and so I agree with you. Although based on prior discussions between Me, Runping, Mac and Milind, it appears to have better fit and more potential with Chukwa framework and so was decided to be part of Chukwa project.
Milind/Mac/Runping, comments? -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12393268/hadoop_vaidya_trunk.patch against trunk revision 709609. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 20 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3525/testReport/ This message is automatically generated. I believe that the right solution is to make this project a stand-alone contrib project and handle the chukwa dependencies dynamically. Otherwise, users that aren't using chukwa won't see this tool.
making a patch available for hadoop-vaidya in the contrib (moved it outside the chukwa as thought initially).
changed the fixed version to 0.20.0 (by mistake was selected 0.19 )
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12394120/hadoop-vaidya-trunk.patch against trunk revision 719431. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 20 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3615/testReport/ This message is automatically generated. Found a bug.. so canceling the patch and will be resubmitting the fixed version soon.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12394453/hadoop_vaidya_0200.patch against trunk revision 719787. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 21 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3636/testReport/ This message is automatically generated. failed unit test is NOT related to this patch.
Sorry, I've only glanced at the patch, but it would be better to make the contrib module 'vaidya' instead of 'hadoop-vaidya' since the hadoop is implied.
New file attached hadoop_vaidya_0200-1.patch with changes making the module named vaidya under contrib (instead of hadoop-vaidya). Corresponding changes also made in the source and documentation.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12394648/hadoop_vaidya_0200-1.patch against trunk revision 720698. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 21 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3653/testReport/ This message is automatically generated. Can we commit this patch? some folks inquiring for it.
I just committed this. Thanks, Suhas!
Integrated in Hadoop-trunk #683 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/683/
). Add Vaidya tool to analyze map/reduce job logs for performanc problems. (Suhas Gogate via omalley) |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
New Feature
Closed
Major
Hadoop-Vaidya : Rule based performance diagnostic tool for Map/Reduce jobs
