Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-967

TaskTracker does not need to fully unjar job jars

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      For efficiency, TaskTrackers no longer unjar the job jar into the job cache directory. Users who previously depended on this functionality for shipping non-code dependencies can use the undocumented configuration parameter "mapreduce.job.jar.unpack.pattern" to cause specific jar contents to be unpacked.
      Show
      For efficiency, TaskTrackers no longer unjar the job jar into the job cache directory. Users who previously depended on this functionality for shipping non-code dependencies can use the undocumented configuration parameter "mapreduce.job.jar.unpack.pattern" to cause specific jar contents to be unpacked.

      Description

      In practice we have seen some users submitting job jars that consist of 10,000+ classes. Unpacking these jars into mapred.local.dir and then cleaning up after them has a significant cost (both in wall clock and in unnecessary heavy disk utilization). This cost can be easily avoided

      1. mapreduce-967.txt
        13 kB
        Todd Lipcon
      2. mapreduce-967.txt
        13 kB
        Todd Lipcon
      3. mapreduce-967.txt
        13 kB
        Todd Lipcon
      4. mapreduce-967.txt
        12 kB
        Todd Lipcon
      5. mapreduce-967.txt
        21 kB
        Todd Lipcon
      6. mapreduce-967.txt
        20 kB
        Todd Lipcon
      7. mapreduce-967-branch-0.20.txt
        4 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          Currently, TaskTracker.localizeJob completely unjars job.jar in jobCacheDir. TaskRunner then appends <jobCacheDir>/classes:<jobCacheDir>/lib/*:<jobCacheDir>/ to the task classpath. Instead, I propose that we only unpack the classes/ and lib/ portions of job.jar, and add <jobCacheDir>/job.jar to the task classpath in lieu of <jobCacheDir>/

          While we're at it, I'm not sure I see the purpose of the "classes/" directory - this is not standard Jar layout by any means, and seems unnecessary. But that issue is orthogonal to this ticket.

          Attaching a preliminary patch against branch-20, though this should go into trunk and probably not the branch. I just want to test this on a real workload first.

          Show
          Todd Lipcon added a comment - Currently, TaskTracker.localizeJob completely unjars job.jar in jobCacheDir. TaskRunner then appends <jobCacheDir>/classes:<jobCacheDir>/lib/*:<jobCacheDir>/ to the task classpath. Instead, I propose that we only unpack the classes/ and lib/ portions of job.jar, and add <jobCacheDir>/job.jar to the task classpath in lieu of <jobCacheDir>/ While we're at it, I'm not sure I see the purpose of the "classes/" directory - this is not standard Jar layout by any means, and seems unnecessary. But that issue is orthogonal to this ticket. Attaching a preliminary patch against branch-20, though this should go into trunk and probably not the branch. I just want to test this on a real workload first.
          Hide
          Todd Lipcon added a comment -

          Tested this manually by unjarring the examples jar, creating a new jar 'lib/sleep.jar' with all of the inner classes of SleepJob, removing them from the outer jar, and jarring it back up. This job ran correctly, and while it was running I investigated jobCache/<job_id>/jars to verify that the classes had not been unpacked, but lib/sleep.jar had been.

          Show
          Todd Lipcon added a comment - Tested this manually by unjarring the examples jar, creating a new jar 'lib/sleep.jar' with all of the inner classes of SleepJob, removing them from the outer jar, and jarring it back up. This job ran correctly, and while it was running I investigated jobCache/<job_id>/jars to verify that the classes had not been unpacked, but lib/sleep.jar had been.
          Hide
          Doug Cutting added a comment -

          > I'm not sure I see the purpose of the "classes/" directory [ ... ]

          This was done by analogy with .war files.

          Show
          Doug Cutting added a comment - > I'm not sure I see the purpose of the "classes/" directory [ ... ] This was done by analogy with .war files.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Now that there are changes to RunJar, for trunk, can you move RunJar to mapreduce from common as is generally desired (https://issues.apache.org/jira/browse/MAPREDUCE-727?focusedCommentId=12728372&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12728372) ?

          Also, I think, it will be good to make the filter to specify the directories/files in job.jar to be un-jarred as configurable. This way we can also maintain backward compatibility to the current scenario where in we un-jar everything. The configuration can be a comma separated list of files/dires for example. You may also need changes to the JarEntryFilter to accept wild-card entries. Thoughts?

          Show
          Vinod Kumar Vavilapalli added a comment - Now that there are changes to RunJar, for trunk, can you move RunJar to mapreduce from common as is generally desired ( https://issues.apache.org/jira/browse/MAPREDUCE-727?focusedCommentId=12728372&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12728372 ) ? Also, I think, it will be good to make the filter to specify the directories/files in job.jar to be un-jarred as configurable . This way we can also maintain backward compatibility to the current scenario where in we un-jar everything. The configuration can be a comma separated list of files/dires for example. You may also need changes to the JarEntryFilter to accept wild-card entries. Thoughts?
          Hide
          Todd Lipcon added a comment -

          Now that there are changes to RunJar, for trunk, can you move RunJar to mapreduce from common

          Sure thing. I'll take care of that when I post the patch for trunk.

          Also, I think, it will be good to make the filter to specify the directories/files in job.jar to be un-jarred as configurable

          Do you see any use for filters here beyond a straight regex? We can express the old behaviour as /./ and the new behavior as /^(lib|classes)\//. Also, I'd prefer to make this an *undocumented configuration parameter, since I think there is very little use for the old version and we don't want to encourage people to abuse this.

          Would you see this being used as a per-job option or a TaskTracker-scoped option?

          Show
          Todd Lipcon added a comment - Now that there are changes to RunJar, for trunk, can you move RunJar to mapreduce from common Sure thing. I'll take care of that when I post the patch for trunk. Also, I think, it will be good to make the filter to specify the directories/files in job.jar to be un-jarred as configurable Do you see any use for filters here beyond a straight regex? We can express the old behaviour as /. / and the new behavior as /^(lib|classes)\//. Also, I'd prefer to make this an *undocumented configuration parameter, since I think there is very little use for the old version and we don't want to encourage people to abuse this. Would you see this being used as a per-job option or a TaskTracker-scoped option?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Now that there are changes to RunJar, for trunk, can you move RunJar to mapreduce from common

          Sure thing. I'll take care of that when I post the patch for trunk.

          Thanks! Please also make sure the new class in mapreduce is in org.apache.hadoop.mapreduce.util and create a deprecate class in org.apache.hadoop.util which uses functionality from org.apache.hadoop.mapreduce.util.RunJar.

          Do you see any use for filters here beyond a straight regex? We can express the old behaviour as /./ and the new behavior as /^(lib|classes)\//.

          Yes, that should do, I think.

          Also, I'd prefer to make this an *undocumented configuration parameter, since I think there is very little use for the old version and we don't want to encourage people to abuse this.

          Agreed. Even now, it is undocumented, AFAIK. A more appropriate reasoning for making it a configuration is that some users may want directories other than lib or classes to be unjarred.

          Would you see this being used as a per-job option or a TaskTracker-scoped option?

          Per-job. By the time, we un-jar stuff on the TT, job configuration is already localized, so it's easy to get this option just before un-jarring.

          Show
          Vinod Kumar Vavilapalli added a comment - Now that there are changes to RunJar, for trunk, can you move RunJar to mapreduce from common Sure thing. I'll take care of that when I post the patch for trunk. Thanks! Please also make sure the new class in mapreduce is in org.apache.hadoop.mapreduce.util and create a deprecate class in org.apache.hadoop.util which uses functionality from org.apache.hadoop.mapreduce.util.RunJar. Do you see any use for filters here beyond a straight regex? We can express the old behaviour as /./ and the new behavior as /^(lib|classes)\//. Yes, that should do, I think. Also, I'd prefer to make this an *undocumented configuration parameter, since I think there is very little use for the old version and we don't want to encourage people to abuse this. Agreed. Even now, it is undocumented, AFAIK. A more appropriate reasoning for making it a configuration is that some users may want directories other than lib or classes to be unjarred. Would you see this being used as a per-job option or a TaskTracker-scoped option? Per-job. By the time, we un-jar stuff on the TT, job configuration is already localized, so it's easy to get this option just before un-jarring.
          Hide
          Todd Lipcon added a comment -

          Sounds good. Thanks for the review, Vinod. I'll take care of this soon and upload an up-to-date patch. I should have results from testing on the cluster as well to see if this does indeed reduce disk utilization appreciably.

          Show
          Todd Lipcon added a comment - Sounds good. Thanks for the review, Vinod. I'll take care of this soon and upload an up-to-date patch. I should have results from testing on the cluster as well to see if this does indeed reduce disk utilization appreciably.
          Hide
          Todd Lipcon added a comment -

          Please also make sure the new class in mapreduce is in org.apache.hadoop.mapreduce.util and create a deprecate class in org.apache.hadoop.util which uses functionality from org.apache.hadoop.mapreduce.util.RunJar.

          Can we do a real dependency in this direction or do I have to use reflection to cross the project boundary? I was under the impression that cyclic dependencies were bad.

          I can certainly mark the existing implementation as Deprecated, but keep the implementation there. Then I can add a non-deprecated shim class in mapreduce.util to wrap the old version with deprecation warnings supressed. We can then do the move for reals next version.

          Sound good?

          Show
          Todd Lipcon added a comment - Please also make sure the new class in mapreduce is in org.apache.hadoop.mapreduce.util and create a deprecate class in org.apache.hadoop.util which uses functionality from org.apache.hadoop.mapreduce.util.RunJar. Can we do a real dependency in this direction or do I have to use reflection to cross the project boundary? I was under the impression that cyclic dependencies were bad. I can certainly mark the existing implementation as Deprecated, but keep the implementation there. Then I can add a non-deprecated shim class in mapreduce.util to wrap the old version with deprecation warnings supressed. We can then do the move for reals next version. Sound good?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Following the way we did this elsewhere (MAPREDUCE-711), we should

          • remove the old class org.apache.hadoop.util.RunJar completely from the common project,
          • move the code completely to a new class org.apache.hadoop.mapreduce.util.RunJar in mapreduce project
          • for supporting deprecation, create a new class in org.apache.hadoop.util.RunJar in mapreduce project which is just a placeholder and redirects all the functionality to the class in org.apache.hadoop.mapreduce.util.RunJar. This class should be deprecated rightaway.
          • change all the framework references now to use org.apache.hadoop.mapreduce.util.RunJar

          This avoids the cyclic dependency that you've mentioned. But it definitely breaks code of those who depend explicitly only on the common jar, but as discussed on MAPREDUCE-711 and in particular (https://issues.apache.org/jira/browse/MAPREDUCE-711?focusedCommentId=12729692&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12729692), project split already broke that kinda dependency.

          Show
          Vinod Kumar Vavilapalli added a comment - Following the way we did this elsewhere ( MAPREDUCE-711 ), we should remove the old class org.apache.hadoop.util.RunJar completely from the common project, move the code completely to a new class org.apache.hadoop.mapreduce.util.RunJar in mapreduce project for supporting deprecation, create a new class in org.apache.hadoop.util.RunJar in mapreduce project which is just a placeholder and redirects all the functionality to the class in org.apache.hadoop.mapreduce.util.RunJar . This class should be deprecated rightaway. change all the framework references now to use org.apache.hadoop.mapreduce.util.RunJar This avoids the cyclic dependency that you've mentioned. But it definitely breaks code of those who depend explicitly only on the common jar, but as discussed on MAPREDUCE-711 and in particular ( https://issues.apache.org/jira/browse/MAPREDUCE-711?focusedCommentId=12729692&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12729692 ), project split already broke that kinda dependency.
          Hide
          Todd Lipcon added a comment -

          One note about this JIRA - it will need some fix for Streaming as well. The common way that people ship scripts for streaming is using the "-file foo.py" argument. This just includes foo.py in the job jar and assumes it will be unpacked on the other side. With this patch, it won't unpack those and breaks the -file argument's primary use case.

          Two options to fix this issue:

          1. We could change -file to use DistributedCache instead. The fact that -file and -files do different things is confusing in the first place, but changing the behavior is potentially breaking change, I think.
          2. We could change Streaming to add all of the -file paths to the new configuration parameter such that the existing behavior is preserved.

          If no one else has a preference I'll go for option #2 above.

          Show
          Todd Lipcon added a comment - One note about this JIRA - it will need some fix for Streaming as well. The common way that people ship scripts for streaming is using the "-file foo.py" argument. This just includes foo.py in the job jar and assumes it will be unpacked on the other side. With this patch, it won't unpack those and breaks the -file argument's primary use case. Two options to fix this issue: We could change -file to use DistributedCache instead. The fact that -file and -files do different things is confusing in the first place, but changing the behavior is potentially breaking change, I think. We could change Streaming to add all of the -file paths to the new configuration parameter such that the existing behavior is preserved. If no one else has a preference I'll go for option #2 above.
          Hide
          Todd Lipcon added a comment -

          Here's a patch against trunk. This relies on HADOOP-6346.

          It includes a new test case for streaming that verifies the fix of the bug reported in my comment above.

          Show
          Todd Lipcon added a comment - Here's a patch against trunk. This relies on HADOOP-6346 . It includes a new test case for streaming that verifies the fix of the bug reported in my comment above.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423639/mapreduce-967.txt
          against trunk revision 831037.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/111/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/111/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/111/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423639/mapreduce-967.txt against trunk revision 831037. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/111/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/111/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/111/console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Hudson failure is because it needs the new core jar from HADOOP-6346

          Show
          Todd Lipcon added a comment - Hudson failure is because it needs the new core jar from HADOOP-6346
          Hide
          Todd Lipcon added a comment -

          Small update to previous patch - forgot to change references to RunJar to point to its new location.

          Show
          Todd Lipcon added a comment - Small update to previous patch - forgot to change references to RunJar to point to its new location.
          Hide
          Todd Lipcon added a comment -

          Linking blocking common ticket.

          Show
          Todd Lipcon added a comment - Linking blocking common ticket.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423791/mapreduce-967.txt
          against trunk revision 831037.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/116/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/116/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/116/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423791/mapreduce-967.txt against trunk revision 831037. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/116/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/116/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/116/console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          One note about this JIRA - it will need some fix for Streaming as well. The common way that people ship scripts for streaming is using the "-file foo.py" argument. This just includes foo.py in the job jar and assumes it will be unpacked on the other side. With this patch, it won't unpack those and breaks the -file argument's primary use case.

          I've just looked up the documentation, and, though not very explicit, -file is part of the job.jar (and hence for small files) whereas -files, -archives can be used for large files. So, going by that, I am +1 for the 2nd approach that you've outlined. If we want to be sure, we can make the above distinction explicit in the forrest docs.

          Will quickly look at your patch.

          Show
          Vinod Kumar Vavilapalli added a comment - One note about this JIRA - it will need some fix for Streaming as well. The common way that people ship scripts for streaming is using the "-file foo.py" argument. This just includes foo.py in the job jar and assumes it will be unpacked on the other side. With this patch, it won't unpack those and breaks the -file argument's primary use case. I've just looked up the documentation, and, though not very explicit, -file is part of the job.jar (and hence for small files) whereas -files, -archives can be used for large files. So, going by that, I am +1 for the 2nd approach that you've outlined. If we want to be sure, we can make the above distinction explicit in the forrest docs. Will quickly look at your patch.
          Hide
          Todd Lipcon added a comment -

          The patch attached above does do option #2 as outlined above, so I think we're OK.

          Show
          Todd Lipcon added a comment - The patch attached above does do option #2 as outlined above, so I think we're OK.
          Hide
          Todd Lipcon added a comment -

          New patch for the new version of HADOOP-6346. This one does not move the RunJar class to mapreduce, since we determined over in that issue that it isn't the best course of action.

          One question for reviewer: the constant for the new configuration key is in JobContext, whereas the default is in JobConf. I was following some other examples from the code, but it seems a little bit messy here. Where are the right places to add new configuration parameters that work in both APIs?

          Show
          Todd Lipcon added a comment - New patch for the new version of HADOOP-6346 . This one does not move the RunJar class to mapreduce, since we determined over in that issue that it isn't the best course of action. One question for reviewer: the constant for the new configuration key is in JobContext, whereas the default is in JobConf. I was following some other examples from the code, but it seems a little bit messy here. Where are the right places to add new configuration parameters that work in both APIs?
          Hide
          Todd Lipcon added a comment -

          Forgot to note - this will probably fail Hudson since it requires the common jar built from HADOOP-6346 to add new functions to support regexes in Configuration.

          Show
          Todd Lipcon added a comment - Forgot to note - this will probably fail Hudson since it requires the common jar built from HADOOP-6346 to add new functions to support regexes in Configuration.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12424909/mapreduce-967.txt
          against trunk revision 835968.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/139/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/139/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/139/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424909/mapreduce-967.txt against trunk revision 835968. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/139/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/139/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/139/console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          HADOOP-6346 is ready to go pending last approval from Hudson. Does anyone have a moment to review this one? You'll need to run against a common jar built from that jira, if you want to try it. Also please note my question asked above. Thanks!

          Show
          Todd Lipcon added a comment - HADOOP-6346 is ready to go pending last approval from Hudson. Does anyone have a moment to review this one? You'll need to run against a common jar built from that jira, if you want to try it. Also please note my question asked above. Thanks!
          Hide
          Tom White added a comment -

          +1 This looks good to me.

          One question for reviewer: the constant for the new configuration key is in JobContext, whereas the default is in JobConf. I was following some other examples from the code, but it seems a little bit messy here. Where are the right places to add new configuration parameters that work in both APIs?

          The key should certainly go in JobContext, but where the default is located is less clear. Defaults tend to be defined in the class that they are used, which is JobConf in this case. However, JobConf is deprecated and will disappear, although it may still be used by the implementation (i.e. not be a part of the public API), in which case what you have done is fine.

          Show
          Tom White added a comment - +1 This looks good to me. One question for reviewer: the constant for the new configuration key is in JobContext, whereas the default is in JobConf. I was following some other examples from the code, but it seems a little bit messy here. Where are the right places to add new configuration parameters that work in both APIs? The key should certainly go in JobContext, but where the default is located is less clear. Defaults tend to be defined in the class that they are used, which is JobConf in this case. However, JobConf is deprecated and will disappear, although it may still be used by the implementation (i.e. not be a part of the public API), in which case what you have done is fine.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Overall, the patch looked good to me too.

          One point of concern, similar to what I had on HADOOP-6346, is the classpath: paths unjarred are not put on the classpath. Granted, files unjarred like this are usable for direct access(like the -file case of streaming). Even if one gives a user specified directory to be unjarred, I don't see a way he/she can use it to find classes. Write a custom ClassLoader? In that case, does he/she even need to unjar it? Any idea?

          Another point: the changes to classpath in TaskRunner seem suspicious. Earlier, we had

              classPaths.add(jobCacheDir.toString());
          

          and now only

              classPaths.add(new File(jobCacheDir, "job.jar").toString());
          

          Is this intentional? May be we need both? Throw some light?

          Show
          Vinod Kumar Vavilapalli added a comment - Overall, the patch looked good to me too. One point of concern, similar to what I had on HADOOP-6346 , is the classpath : paths unjarred are not put on the classpath. Granted, files unjarred like this are usable for direct access(like the -file case of streaming). Even if one gives a user specified directory to be unjarred, I don't see a way he/she can use it to find classes. Write a custom ClassLoader ? In that case, does he/she even need to unjar it? Any idea? Another point: the changes to classpath in TaskRunner seem suspicious. Earlier, we had classPaths.add(jobCacheDir.toString()); and now only classPaths.add( new File(jobCacheDir, "job.jar" ).toString()); Is this intentional? May be we need both? Throw some light?
          Hide
          Todd Lipcon added a comment -

          I think it used to add the whole jobCacheDir mostly out of sloppiness. The options for getting classes are:

          1. Put it in the job.jar
          2. Put it in a 'classes/' directory inside the job jar
          3. Put it in its own jar, which is inside a lib/ directory in the job jar

          I can't think of any reasons why you'd need any more options than this - the first and second option are already redundant. If someone has a really bizarre use case, they do have access to the job cache dir path through the job configuration, so yes, I'd recommend a custom classloader to them. Unless we know of an existing app that can't be satisfied by one of the above, I think it's better to clean up the classpath rather than continue to put the job cache dir on it when it seems unnecessary (at least for every job I've ever seen).

          Show
          Todd Lipcon added a comment - I think it used to add the whole jobCacheDir mostly out of sloppiness. The options for getting classes are: Put it in the job.jar Put it in a 'classes/' directory inside the job jar Put it in its own jar, which is inside a lib/ directory in the job jar I can't think of any reasons why you'd need any more options than this - the first and second option are already redundant. If someone has a really bizarre use case, they do have access to the job cache dir path through the job configuration, so yes, I'd recommend a custom classloader to them. Unless we know of an existing app that can't be satisfied by one of the above, I think it's better to clean up the classpath rather than continue to put the job cache dir on it when it seems unnecessary (at least for every job I've ever seen).
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Agreed. Even I was at a loss of how flexible we should be. Till far, AFAIK, this whole "job.jar's unjarring and putting in classpath" business is nowhere clearly documented. The only documentation I could find is (from mapred_tutorial) the following, which clearly is unclear about the above:

          ${mapred.local.dir}/taskTracker/jobcache/$jobid/jars/  : The jars directory, which has the job jar file
          and expanded jar. The job.jar is the application's jar file that is automatically distributed to each machine.
          It is expanded in jars directory before the tasks for the job start. The job.jar location is accessible to the
          application through the api  JobConf.getJar() . To access the unjarred directory, JobConf.getJar().getParent()
          can be called.
          
          • One take away is that along with the other changes in this JIRA we should definitely document clearly the points you've mentioned above w.r.t the classpath.
          • The second point is that your change of classpath to no more include jobCacheDir makes this JIRA issue an incompatible change. If we wish to maintain the backward compatibility, I suggest we add both job.jar as well as the jobCacheDir to the classpath. The reason I am nit-picky about this is things like these have the potential to come back and catch us unawares in the future.
          Show
          Vinod Kumar Vavilapalli added a comment - Agreed. Even I was at a loss of how flexible we should be. Till far, AFAIK, this whole "job.jar's unjarring and putting in classpath" business is nowhere clearly documented. The only documentation I could find is (from mapred_tutorial) the following, which clearly is unclear about the above: ${mapred.local.dir}/taskTracker/jobcache/$jobid/jars/ : The jars directory, which has the job jar file and expanded jar. The job.jar is the application's jar file that is automatically distributed to each machine. It is expanded in jars directory before the tasks for the job start. The job.jar location is accessible to the application through the api JobConf.getJar() . To access the unjarred directory, JobConf.getJar().getParent() can be called. One take away is that along with the other changes in this JIRA we should definitely document clearly the points you've mentioned above w.r.t the classpath. The second point is that your change of classpath to no more include jobCacheDir makes this JIRA issue an incompatible change. If we wish to maintain the backward compatibility, I suggest we add both job.jar as well as the jobCacheDir to the classpath. The reason I am nit-picky about this is things like these have the potential to come back and catch us unawares in the future.
          Hide
          Todd Lipcon added a comment -

          we should definitely document clearly the points you've mentioned above w.r.t the classpath.

          You're totally right, and I actually did this and forgot to upload the patch! My bad. Here's a new one.

          makes this JIRA issue an incompatible change

          Yes, this is technically incompatible. But I think it's not a problem for the following reasons:

          • Since job.jar is itself added to the classpath, the standard classloader will pick up anything inside job.jar just as if it were expanded and the resulting dir were put on the classpath
          • The only other people this should break are those who are using java.io (or other non-classpath-related access methods) to access things unpacked from the jar. The new configuration parameter is a suitable workaround for them (as demonstrated by Streaming). In this case, what's on the classpath doesn't matter since they're not using a ClassLoader anyhow.
          • Non-java applications are the only ones for whom the above two points don't apply, but non-Java applications don't have any concept of classpath and therefore it shouldn't be a problem.

          Philosophically, isn't pre-1.0 exactly when we should be making these minor incompatible changes for the purposes of code cleanliness? Compared to the other drastic changes we're putting in 22, this is hardly a showstopper. I don't see anything against the change you're requesting, except that I think we should do everything in our power now to clean up the code before we call Hadoop 1.0. If I'm the only one with this philosophy, I'll acquiesce, but I think the sloppy classpath is just as likely to come back to bite us as fixing it.

          Show
          Todd Lipcon added a comment - we should definitely document clearly the points you've mentioned above w.r.t the classpath. You're totally right, and I actually did this and forgot to upload the patch! My bad. Here's a new one. makes this JIRA issue an incompatible change Yes, this is technically incompatible. But I think it's not a problem for the following reasons: Since job.jar is itself added to the classpath, the standard classloader will pick up anything inside job.jar just as if it were expanded and the resulting dir were put on the classpath The only other people this should break are those who are using java.io (or other non-classpath-related access methods) to access things unpacked from the jar. The new configuration parameter is a suitable workaround for them (as demonstrated by Streaming). In this case, what's on the classpath doesn't matter since they're not using a ClassLoader anyhow. Non-java applications are the only ones for whom the above two points don't apply, but non-Java applications don't have any concept of classpath and therefore it shouldn't be a problem. Philosophically, isn't pre-1.0 exactly when we should be making these minor incompatible changes for the purposes of code cleanliness? Compared to the other drastic changes we're putting in 22, this is hardly a showstopper. I don't see anything against the change you're requesting, except that I think we should do everything in our power now to clean up the code before we call Hadoop 1.0. If I'm the only one with this philosophy, I'll acquiesce, but I think the sloppy classpath is just as likely to come back to bite us as fixing it.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427302/mapreduce-967.txt
          against trunk revision 888269.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/302/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/302/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/302/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427302/mapreduce-967.txt against trunk revision 888269. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/302/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/302/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/302/console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Your argument w.r.t the classpath (points 1 and 3) sounds reasonable. The newly added documentation also mitigates some of my previous issues. Still, I think that with the default value for the configuration property as (lib|classes) and not (.*), we need to mark this as incompatible and the release note should clearly reflect the breakage of the streaming like use-cases(your 2nd point above) and the available work-around(the configuration property) as to how to get everything unjarred.

          Given that, the patch looks good to me. +1, pending commit of HADOOP-6346.

          Show
          Vinod Kumar Vavilapalli added a comment - Your argument w.r.t the classpath (points 1 and 3) sounds reasonable. The newly added documentation also mitigates some of my previous issues. Still, I think that with the default value for the configuration property as (lib|classes) and not (.*), we need to mark this as incompatible and the release note should clearly reflect the breakage of the streaming like use-cases(your 2nd point above) and the available work-around(the configuration property) as to how to get everything unjarred. Given that, the patch looks good to me. +1, pending commit of HADOOP-6346 .
          Hide
          Todd Lipcon added a comment -

          Retriggering hudson now that HADOOP-6346 was committed.

          Show
          Todd Lipcon added a comment - Retriggering hudson now that HADOOP-6346 was committed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427302/mapreduce-967.txt
          against trunk revision 888761.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/313/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/313/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/313/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/313/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427302/mapreduce-967.txt against trunk revision 888761. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/313/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/313/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/313/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/313/console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Fixed patch - the previous one worked with a slightly old version of the common patch it depended on. Apologies for not re-running the tests locally before submitting to Hudson yesterday.

          Ran the tests on this fixed patch overnight and looked good here - will resubmit to Hudson to confirm.

          Show
          Todd Lipcon added a comment - Fixed patch - the previous one worked with a slightly old version of the common patch it depended on. Apologies for not re-running the tests locally before submitting to Hudson yesterday. Ran the tests on this fixed patch overnight and looked good here - will resubmit to Hudson to confirm.
          Hide
          Todd Lipcon added a comment -

          Ahh, forgot --no-prefix arg on the diff. Sorry for the spam.

          Show
          Todd Lipcon added a comment - Ahh, forgot --no-prefix arg on the diff. Sorry for the spam.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427634/mapreduce-967.txt
          against trunk revision 889085.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/316/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/316/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/316/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/316/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427634/mapreduce-967.txt against trunk revision 889085. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/316/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/316/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/316/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/316/console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          The failing streaming tests don't fail on my local box. I think it might be MAPREDUCE-1275 at fault here... since this patch does affect classpaths and streaming, I'm toggling patch status to give hudson a second go.

          Show
          Todd Lipcon added a comment - The failing streaming tests don't fail on my local box. I think it might be MAPREDUCE-1275 at fault here... since this patch does affect classpaths and streaming, I'm toggling patch status to give hudson a second go.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427634/mapreduce-967.txt
          against trunk revision 889486.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/186/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/186/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/186/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/186/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427634/mapreduce-967.txt against trunk revision 889486. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/186/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/186/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/186/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/186/console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          hudson take three (last failure was some bizarre hudson communication error between the Hudson executor and master, I think?)

          Show
          Todd Lipcon added a comment - hudson take three (last failure was some bizarre hudson communication error between the Hudson executor and master, I think?)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427634/mapreduce-967.txt
          against trunk revision 889571.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/321/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/321/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/321/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/321/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427634/mapreduce-967.txt against trunk revision 889571. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/321/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/321/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/321/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/321/console This message is automatically generated.
          Hide
          Tom White added a comment -

          I've just committed this. Thanks Todd!

          (I ran the tests that had failed locally and they all passed.)

          Show
          Tom White added a comment - I've just committed this. Thanks Todd! (I ran the tests that had failed locally and they all passed.)
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/ )

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development