Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1131

Using profilers other than hprof can cause JobClient to report job failure

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If task profiling is enabled, the JobClient will download the profile.out file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.

      1. MAPREDUCE-1131.patch
        7 kB
        Aaron Kimball
      2. MAPREDUCE-1131.6.patch
        7 kB
        Aaron Kimball
      3. MAPREDUCE-1131.5.patch
        4 kB
        Aaron Kimball
      4. MAPREDUCE-1131.4.patch
        4 kB
        Aaron Kimball
      5. MAPREDUCE-1131.3.patch
        8 kB
        Aaron Kimball
      6. MAPREDUCE-1131.2.patch
        8 kB
        Aaron Kimball

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/ )
          Hide
          Tom White added a comment -

          I've just committed this. Thanks Aaron!

          Show
          Tom White added a comment - I've just committed this. Thanks Aaron!
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Latest patch looks good to me. +1.

          Show
          Vinod Kumar Vavilapalli added a comment - Latest patch looks good to me. +1.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427970/MAPREDUCE-1131.6.patch
          against trunk revision 890531.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427970/MAPREDUCE-1131.6.patch against trunk revision 890531. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/console This message is automatically generated.
          Hide
          Aaron Kimball added a comment -

          New patch with tests updated per Vinod's request.

          Ran test-patch locally:

               [exec] +1 overall.- - 
               [exec]- 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]- 
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec]- 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]- 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]- 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec]- 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec]- 
          
          Show
          Aaron Kimball added a comment - New patch with tests updated per Vinod's request. Ran test-patch locally: [exec] +1 overall.- - [exec]- [exec] +1 @author. The patch does not contain any @author tags. [exec]- [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec]- [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec]- [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec]- [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec]- [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec]-
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Also, now that the patch needs some update, can you also add a positive test to verify success with hprof? We can then also verify that profile files named like $attempt-id.profile are indeed download by the JobClient when hprof(by default) is used.

          Show
          Vinod Kumar Vavilapalli added a comment - Also, now that the patch needs some update, can you also add a positive test to verify success with hprof? We can then also verify that profile files named like $attempt-id.profile are indeed download by the JobClient when hprof(by default) is used.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Patch changes look good to me.

          One issue with the testcase - because now we are passing an invalid agentlib, tasks themselves are failing in the testcase, as can be expected, and so the test takes long time to finish (the new testcase itself takes more than 3mins). Doing the following makes the test run very fast(~30secs):

          job.setMaxMapAttempts(1);
          job.setMaxReduceAttempts(1);
          job.setJobSetupCleanupNeeded(false);
          

          Also, w.r.t TestMRJobClient, can you please remove the unused import of FileNotFoundException and also convert the test into a junit 4 test?

          Show
          Vinod Kumar Vavilapalli added a comment - Patch changes look good to me. One issue with the testcase - because now we are passing an invalid agentlib, tasks themselves are failing in the testcase, as can be expected, and so the test takes long time to finish (the new testcase itself takes more than 3mins). Doing the following makes the test run very fast(~30secs): job.setMaxMapAttempts(1); job.setMaxReduceAttempts(1); job.setJobSetupCleanupNeeded( false ); Also, w.r.t TestMRJobClient , can you please remove the unused import of FileNotFoundException and also convert the test into a junit 4 test?
          Hide
          Aaron Kimball added a comment -

          Todd, Vinod –

          Do you have thoughts about this version of the patch?

          Show
          Aaron Kimball added a comment - Todd, Vinod – Do you have thoughts about this version of the patch?
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12424426/MAPREDUCE-1131.5.patch
          against trunk revision 834284.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/131/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/131/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/131/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/131/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424426/MAPREDUCE-1131.5.patch against trunk revision 834284. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/131/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/131/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/131/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/131/console This message is automatically generated.
          Hide
          Aaron Kimball added a comment -

          A more correct approach occurred to me 30 seconds after posting #4. Here you go.

          Show
          Aaron Kimball added a comment - A more correct approach occurred to me 30 seconds after posting #4. Here you go.
          Hide
          Aaron Kimball added a comment -

          New patch, incorporates CR comments.

          Show
          Aaron Kimball added a comment - New patch, incorporates CR comments.
          Hide
          Todd Lipcon added a comment -

          Sounds good to me. I'd say split on whitespace, then check part.startsWith(...) for those strings rather than straight string compare - still simple enough and a little more correct.

          Show
          Todd Lipcon added a comment - Sounds good to me. I'd say split on whitespace, then check part.startsWith(...) for those strings rather than straight string compare - still simple enough and a little more correct.
          Hide
          Aaron Kimball added a comment -

          Looking at http://java.sun.com/developer/technicalArticles/Programming/HPROF.html, I think I'm just going to parse the output of conf.getProfileParams() to see if it contains the strings -agentlib:hprof or -Xrunhprof. This is likely not a 100% solution, but would cover an awful lot of uses...

          Show
          Aaron Kimball added a comment - Looking at http://java.sun.com/developer/technicalArticles/Programming/HPROF.html , I think I'm just going to parse the output of conf.getProfileParams() to see if it contains the strings -agentlib:hprof or -Xrunhprof . This is likely not a 100% solution, but would cover an awful lot of uses...
          Hide
          Todd Lipcon added a comment -

          The idea of parsing the params seems reasonable enough, but I'm a little nervous about it. Do we know well the specification for java command line options? Can you run multiple profilers at once with two -javaagent params? [I'm not sure, just wondering if it's going to be just a quick regex vs something more complicated]

          Show
          Todd Lipcon added a comment - The idea of parsing the params seems reasonable enough, but I'm a little nervous about it. Do we know well the specification for java command line options? Can you run multiple profilers at once with two -javaagent params? [I'm not sure, just wondering if it's going to be just a quick regex vs something more complicated]
          Hide
          Aaron Kimball added a comment -

          Vinod,

          Seems reasonable to me. YourKit generates a file in a separate location which developers go and collect manually. I don't have experience with any other profiling tools, so I don't know how many of them do/don't respect a file= parameter like hprof does.

          Looking at its documentation, jprof seems to generate several files in the working directory, but specifying the filename of any one of them is a pain. (It's got a lot of file prefix/suffix arguments.) So it's unlikely that jprof would work with the downloadProfile() as-written anyway. I don't know what other profiling tools are commonly used, so I don't know what else to check.

          So are we sold on "check for hprof and file=%s" arguments, and select downloadProfile() based solely on that?

          • Aaron
          Show
          Aaron Kimball added a comment - Vinod, Seems reasonable to me. YourKit generates a file in a separate location which developers go and collect manually. I don't have experience with any other profiling tools, so I don't know how many of them do/don't respect a file= parameter like hprof does. Looking at its documentation, jprof seems to generate several files in the working directory, but specifying the filename of any one of them is a pain. (It's got a lot of file prefix/suffix arguments.) So it's unlikely that jprof would work with the downloadProfile() as-written anyway. I don't know what other profiling tools are commonly used, so I don't know what else to check. So are we sold on "check for hprof and file=%s " arguments, and select downloadProfile() based solely on that? Aaron
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Sorry for coming late on this. But I think the profile file is originally intended only when hprof is in use, and that to when the 'file' argument is passed .So, instead of adding another configuration, why not downloadProfile() only when hprof is in use and only when the file argument is non-null? This includes parsing the profile-params configuration, of course.

          Or do you expect such file to be generated with other profilers too?

          Show
          Vinod Kumar Vavilapalli added a comment - Sorry for coming late on this. But I think the profile file is originally intended only when hprof is in use, and that to when the 'file' argument is passed .So, instead of adding another configuration, why not downloadProfile() only when hprof is in use and only when the file argument is non-null? This includes parsing the profile-params configuration, of course. Or do you expect such file to be generated with other profilers too?
          Hide
          Aaron Kimball added a comment -

          Failure is unrelated.

          Show
          Aaron Kimball added a comment - Failure is unrelated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12422866/MAPREDUCE-1131.3.patch
          against trunk revision 828253.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/88/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/88/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/88/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/88/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422866/MAPREDUCE-1131.3.patch against trunk revision 828253. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/88/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/88/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/88/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/88/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12422863/MAPREDUCE-1131.2.patch
          against trunk revision 828081.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/199/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/199/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/199/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/199/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422863/MAPREDUCE-1131.2.patch against trunk revision 828081. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/199/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/199/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/199/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/199/console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          +1 lgtm

          Show
          Todd Lipcon added a comment - +1 lgtm
          Hide
          Aaron Kimball added a comment -

          Job name changed.

          Show
          Aaron Kimball added a comment - Job name changed.
          Hide
          Todd Lipcon added a comment -

          Oops - I was thrown by the job name being the same in both cases. You should change the job name for the second submission, I think.

          Show
          Todd Lipcon added a comment - Oops - I was thrown by the job name being the same in both cases. You should change the job name for the second submission, I think.
          Hide
          Aaron Kimball added a comment -

          Todd: the test proves both the positive and the negative forms of the issue. Only the second iteration calls job.setProfileFetchEnabled(false);.

          Show
          Aaron Kimball added a comment - Todd: the test proves both the positive and the negative forms of the issue. Only the second iteration calls job.setProfileFetchEnabled(false); .
          Hide
          Aaron Kimball added a comment -

          Oops. Forgot to ant clean before testing. New patch.

          Show
          Aaron Kimball added a comment - Oops. Forgot to ant clean before testing. New patch.
          Hide
          Todd Lipcon added a comment -

          I'm confused by the test - it seems to submit the same job twice and expect success for one and failure for the other. Did you upload the wrong copy of the test case or something?

          Show
          Todd Lipcon added a comment - I'm confused by the test - it seems to submit the same job twice and expect success for one and failure for the other. Did you upload the wrong copy of the test case or something?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12422859/MAPREDUCE-1131.patch
          against trunk revision 828081.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/198/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/198/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/198/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422859/MAPREDUCE-1131.patch against trunk revision 828081. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/198/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/198/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/198/console This message is automatically generated.
          Hide
          Aaron Kimball added a comment -

          This patch adds a parameter (mapreduce.task.profile.enable.fetch) which defaults to true; if set true, profiler outputs are downloaded by the client. If false, the client does not attempt to fetch the output files. Users of other profilers can disable local fetches so that profiling does not interfere with job success reporting.

          This adds a testcase which runs a job on a MiniMR cluster with profiling enabled, but an empty agentlib string. This causes no output to be generated by a profiler. With fetching disabled, this still allows the job to succeed.

          Show
          Aaron Kimball added a comment - This patch adds a parameter ( mapreduce.task.profile.enable.fetch ) which defaults to true; if set true, profiler outputs are downloaded by the client. If false, the client does not attempt to fetch the output files. Users of other profilers can disable local fetches so that profiling does not interfere with job success reporting. This adds a testcase which runs a job on a MiniMR cluster with profiling enabled, but an empty agentlib string. This causes no output to be generated by a profiler. With fetching disabled, this still allows the job to succeed.

            People

            • Assignee:
              Aaron Kimball
              Reporter:
              Aaron Kimball
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development