Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-463

The job setup and cleanup tasks should be optional

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Added Configuration property "mapred.committer.job.setup.cleanup.needed" to specify whether job-setup and job-cleanup is needed for the job output committer. The default value is true.
      Added Job.setJobSetupCleanupNeeded and JobContext.getJobSetupCleanupNeeded api for setting/getting the configuration.
      If the configuration is set to false, no setup or cleanup will be done.
      Show
      Added Configuration property "mapred.committer.job.setup.cleanup.needed" to specify whether job-setup and job-cleanup is needed for the job output committer. The default value is true. Added Job.setJobSetupCleanupNeeded and JobContext.getJobSetupCleanupNeeded api for setting/getting the configuration. If the configuration is set to false, no setup or cleanup will be done.

      Description

      For jobs that require low latency and do not require setup or cleanup tasks for the job, it should be possible to turn them off for that job.

      1. patch-463.txt
        32 kB
        Amareshwari Sriramadasu
      2. patch-463-1.txt
        32 kB
        Amareshwari Sriramadasu
      3. patch-5785.txt
        25 kB
        Amareshwari Sriramadasu
      4. patch-5785-1.txt
        25 kB
        Amareshwari Sriramadasu
      5. patch-5785-2.txt
        26 kB
        Amareshwari Sriramadasu
      6. patch-5785-3.txt
        28 kB
        Amareshwari Sriramadasu
      7. patch-5785-4.txt
        32 kB
        Amareshwari Sriramadasu
      8. patch-5785-5.txt
        32 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Hide
          Amareshwari Sriramadasu added a comment -

          Should we have a configuration property to speficy whether job setup and job cleanup are needed or not?

          If the configuration is set, there will be no user code executed. JIP will create temporary output directory in initTasks() and delete the same in garbageCollect() (same code as pre-3150). Thoughts?

          Should the configuration be job-level or cluster-level?

          Show
          Amareshwari Sriramadasu added a comment - Should we have a configuration property to speficy whether job setup and job cleanup are needed or not? If the configuration is set, there will be no user code executed. JIP will create temporary output directory in initTasks() and delete the same in garbageCollect() (same code as pre-3150). Thoughts? Should the configuration be job-level or cluster-level?
          Hide
          Devaraj Das added a comment -

          Should the configuration be job-level or cluster-level?

          I vote for cluster-level.

          Show
          Devaraj Das added a comment - Should the configuration be job-level or cluster-level? I vote for cluster-level.
          Hide
          Devaraj Das added a comment -

          Justification why it should be cluster-level config: (1) It is probably something that would conflict with the model of having setup/cleanup tasks in the first place, and hence should be applicable for all jobs or none. (2) It is for latency sensitive apps, and such jobs are usually benchmarks (like terasort) for which we probably will have a dedicated cluster, where we would run only such jobs (3) Might add to confusion when it comes to debugging jobs or the cluster itself.
          The above is my opinion.. Open for discussion. I don't have a very strong preference for one over the other.

          Show
          Devaraj Das added a comment - Justification why it should be cluster-level config: (1) It is probably something that would conflict with the model of having setup/cleanup tasks in the first place, and hence should be applicable for all jobs or none. (2) It is for latency sensitive apps, and such jobs are usually benchmarks (like terasort) for which we probably will have a dedicated cluster, where we would run only such jobs (3) Might add to confusion when it comes to debugging jobs or the cluster itself. The above is my opinion.. Open for discussion. I don't have a very strong preference for one over the other.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          i think it should be job specific as we specify setup/cleanup via OutputCommitter which itself is job specific.

          Also, instead of configuration, as JobSetup and JobCleanup are specified via the job's OutPutCommitter, why not have something in the lines of the following and use them generically to determine if a job needs setup/cleanup.

          OutputCommitter.isJobSetupNeeded()
          OutputCommitter.isJobCleanupNeeded()
          

          Thoughts?

          Show
          Vinod Kumar Vavilapalli added a comment - i think it should be job specific as we specify setup/cleanup via OutputCommitter which itself is job specific. Also, instead of configuration, as JobSetup and JobCleanup are specified via the job's OutPutCommitter, why not have something in the lines of the following and use them generically to determine if a job needs setup/cleanup. OutputCommitter.isJobSetupNeeded() OutputCommitter.isJobCleanupNeeded() Thoughts?
          Hide
          Amareshwari Sriramadasu added a comment -

          Also, instead of configuration, as JobSetup and JobCleanup are specified via the job's OutPutCommitter, why not have something in the lines of the following and use them generically to determine if a job needs setup/cleanup.

          This cannot be done, because it is again user code. and we don't want to run any user code in JobTracker.

          Show
          Amareshwari Sriramadasu added a comment - Also, instead of configuration, as JobSetup and JobCleanup are specified via the job's OutPutCommitter, why not have something in the lines of the following and use them generically to determine if a job needs setup/cleanup. This cannot be done, because it is again user code. and we don't want to run any user code in JobTracker.
          Hide
          He Yongqiang added a comment -

          This is absolutely what i have wanted for a long time. Great! I vote for job-level.

          Show
          He Yongqiang added a comment - This is absolutely what i have wanted for a long time. Great! I vote for job-level.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          This cannot be done, because it is again user code. and we don't want to run any user code in JobTracker.

          If not a method, it can be a simple protected boolean OutputCommitter.isJobSetupNeeded over-ridable by subclasses. Does that work?

          Show
          Vinod Kumar Vavilapalli added a comment - This cannot be done, because it is again user code. and we don't want to run any user code in JobTracker. If not a method, it can be a simple protected boolean OutputCommitter.isJobSetupNeeded over-ridable by subclasses. Does that work?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          If not a method, it can be a simple protected boolean OutputCommitter.isJobSetupNeeded over-ridable by subclasses. Does that work?

          I take that back. Even reading this variable on the JobTracker needs the class to be loaded which in turn means running user code.

          So, +1 for the initial proposal for a configuration property. +1 for job-level configuration.

          Show
          Vinod Kumar Vavilapalli added a comment - If not a method, it can be a simple protected boolean OutputCommitter.isJobSetupNeeded over-ridable by subclasses. Does that work? I take that back. Even reading this variable on the JobTracker needs the class to be loaded which in turn means running user code. So, +1 for the initial proposal for a configuration property. +1 for job-level configuration.
          Hide
          Amareshwari Sriramadasu added a comment -

          Attaching patch for review.

          Patch does the following:
          Reads the configuration mapred.committer.job.setup.cleanup.needed (default value is true). If the configuration is set to false. JobInProgress.initTasks does not initialize setup and cleanup tasks, creates _temporary directory by calling FileOutputCommitter.setupJob(); also changes job state to Running. The directory gets removed in garbageCollect() method.
          To inform state change after initTasks, initJob method is added in TaskTrackerManager (implemented by JobTracker). JT.initJob calls JIP.initTasks without JobTracker lock. Once initialization completes/fails, if there is state change. it locks JobTracker and informs listeners.

          Show
          Amareshwari Sriramadasu added a comment - Attaching patch for review. Patch does the following: Reads the configuration mapred.committer.job.setup.cleanup.needed (default value is true). If the configuration is set to false. JobInProgress.initTasks does not initialize setup and cleanup tasks, creates _temporary directory by calling FileOutputCommitter.setupJob(); also changes job state to Running. The directory gets removed in garbageCollect() method. To inform state change after initTasks, initJob method is added in TaskTrackerManager (implemented by JobTracker). JT.initJob calls JIP.initTasks without JobTracker lock. Once initialization completes/fails, if there is state change. it locks JobTracker and informs listeners.
          Hide
          Amareshwari Sriramadasu added a comment -

          Added the temporary directory to be removed in garvageCollect() to cleanupQueue.

          Show
          Amareshwari Sriramadasu added a comment - Added the temporary directory to be removed in garvageCollect() to cleanupQueue.
          Hide
          Amareshwari Sriramadasu added a comment -

          test patch and ant test passed on my machine.

          Show
          Amareshwari Sriramadasu added a comment - test patch and ant test passed on my machine.
          Hide
          Amar Kamat added a comment -

          Some comments on JobInProgress :
          1. remove JobContext. construct during initTasks.
          2. Move setup/cleanup tasks initialization to different method
          3. introduce methods setupInline/cleanupInline to do setup/cleanup inline in initTasks and garbageCollect.
          4. Remove jobSetupCleanupNeeded guards from canLaunchJobCleanup/SetupTask() methods. And Use them in completedTask whenever maps/reduces complete.
          5. add an api to FileoutputCommitter to give temp dir.

          Show
          Amar Kamat added a comment - Some comments on JobInProgress : 1. remove JobContext. construct during initTasks. 2. Move setup/cleanup tasks initialization to different method 3. introduce methods setupInline/cleanupInline to do setup/cleanup inline in initTasks and garbageCollect. 4. Remove jobSetupCleanupNeeded guards from canLaunchJobCleanup/SetupTask() methods. And Use them in completedTask whenever maps/reduces complete. 5. add an api to FileoutputCommitter to give temp dir.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12409758/patch-5785-1.txt
          against trunk revision 781968.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 18 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/467/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12409758/patch-5785-1.txt against trunk revision 781968. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/467/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch incorporating review comments.

          Show
          Amareshwari Sriramadasu added a comment - Patch incorporating review comments.
          Hide
          Amareshwari Sriramadasu added a comment -

          test patch and ant test passed on my machine

          Show
          Amareshwari Sriramadasu added a comment - test patch and ant test passed on my machine
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12410205/patch-5785-2.txt
          against trunk revision 782881.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 18 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/485/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/485/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/485/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/485/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12410205/patch-5785-2.txt against trunk revision 782881. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/485/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/485/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/485/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/485/console This message is automatically generated.
          Hide
          Sreekanth Ramakrishnan added a comment -

          I tested the patch with respect to JobInitializationPoller. The changes look fine to me.

          Show
          Sreekanth Ramakrishnan added a comment - I tested the patch with respect to JobInitializationPoller . The changes look fine to me.
          Hide
          Amareshwari Sriramadasu added a comment -

          Updated patch with trunk.

          Show
          Amareshwari Sriramadasu added a comment - Updated patch with trunk.
          Hide
          Amar Kamat added a comment -

          Changes look fine to me.

          Show
          Amar Kamat added a comment - Changes look fine to me.
          Hide
          Amareshwari Sriramadasu added a comment -

          test patch and ant test passed on my machine with latest patch.

          Show
          Amareshwari Sriramadasu added a comment - test patch and ant test passed on my machine with latest patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12410301/patch-5785-3.txt
          against trunk revision 784042.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 24 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/495/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/495/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/495/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/495/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12410301/patch-5785-3.txt against trunk revision 784042. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/495/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/495/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/495/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/495/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          Test failures are not related to the patch. All tests passed on my machine.

          Show
          Amareshwari Sriramadasu added a comment - Test failures are not related to the patch. All tests passed on my machine.
          Hide
          Owen O'Malley added a comment -

          You should define a setter and getter for the variable. In particular, it should go into the new Job and JobContext, respectively.

          If the variable is set, no initialization or cleanup should be done. The assumption is that the application needs no setup or cleanup or it will be handled externally from something like Pig, Oozie, or Hive.

          Show
          Owen O'Malley added a comment - You should define a setter and getter for the variable. In particular, it should go into the new Job and JobContext, respectively. If the variable is set, no initialization or cleanup should be done. The assumption is that the application needs no setup or cleanup or it will be handled externally from something like Pig, Oozie, or Hive.
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch incorporating Owen's comments.
          Added Job.setJobSetupCleanupNeeded and JobContext.getJobSetupCleanupNeeded.
          Removed doSetupInline and doCleanupInline.
          Added testcase with new api.

          Show
          Amareshwari Sriramadasu added a comment - Patch incorporating Owen's comments. Added Job.setJobSetupCleanupNeeded and JobContext.getJobSetupCleanupNeeded. Removed doSetupInline and doCleanupInline. Added testcase with new api.
          Hide
          Amareshwari Sriramadasu added a comment -

          attaching correct file

          Show
          Amareshwari Sriramadasu added a comment - attaching correct file
          Hide
          Amareshwari Sriramadasu added a comment -

          Updated patch with trunk.
          Fixed a test failure with earlier patch

          Show
          Amareshwari Sriramadasu added a comment - Updated patch with trunk. Fixed a test failure with earlier patch
          Hide
          Amareshwari Sriramadasu added a comment -

          test-patch and ant test passed on my machine

          Show
          Amareshwari Sriramadasu added a comment - test-patch and ant test passed on my machine
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12411048/patch-5785-5.txt
          against trunk revision 785928.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 27 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          -1 release audit. The applied patch generated 496 release audit warnings (more than the trunk's current 495 warnings).

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/530/testReport/
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/530/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/530/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/530/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/530/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12411048/patch-5785-5.txt against trunk revision 785928. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 release audit. The applied patch generated 496 release audit warnings (more than the trunk's current 495 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/530/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/530/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/530/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/530/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/530/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          -1 release audit. looks wrong. All the newly added files have license header. releaseAuditDiffWarnings.txt shows org.apache.hadoop.mapreduce.JobContext.html in the diff, which is not touched.
          -1 core tests. Test failure is not related to the patch. Raised HADOOP-6086. Test passed on my machine
          -1 contrib tests. Failure is a known issue.

          Show
          Amareshwari Sriramadasu added a comment - -1 release audit. looks wrong. All the newly added files have license header. releaseAuditDiffWarnings.txt shows org.apache.hadoop.mapreduce.JobContext.html in the diff, which is not touched. -1 core tests. Test failure is not related to the patch. Raised HADOOP-6086 . Test passed on my machine -1 contrib tests. Failure is a known issue.
          Hide
          Amar Kamat added a comment -

          patch looks fine to me.

          Show
          Amar Kamat added a comment - patch looks fine to me.
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch updated with new mared directory structure.

          Show
          Amareshwari Sriramadasu added a comment - Patch updated with new mared directory structure.
          Hide
          Amareshwari Sriramadasu added a comment -

          test-patch result :

               [exec]
               [exec] +1 overall.
               [exec]
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]
               [exec]     +1 tests included.  The patch appears to include 27 new or modified tests.
               [exec]
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec]
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec]
               [exec]
          

          run-test-mapred and test-contrib passed on my machine.

          Show
          Amareshwari Sriramadasu added a comment - test-patch result : [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 27 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] run-test-mapred and test-contrib passed on my machine.
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch updated with trunk.

          test-patch and ant test passed on my machine.

          Show
          Amareshwari Sriramadasu added a comment - Patch updated with trunk. test-patch and ant test passed on my machine.
          Hide
          Sharad Agarwal added a comment -

          I committed this. Thanks Amareshwari!

          Show
          Sharad Agarwal added a comment - I committed this. Thanks Amareshwari!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #15 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/15/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #15 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/15/ )
          Hide
          yuling added a comment -

          in patch patch-5785-1.txt, _temporary will be created by this code:
          OutputCommitter committer = new FileOutputCommitter();
          committer.setupJob(jobContext);
          but it was deleted from patch-5785-2.txt, thus _temporary won't be created if the variable is set. Can anybody tell me why ?

          Show
          yuling added a comment - in patch patch-5785-1.txt, _temporary will be created by this code: OutputCommitter committer = new FileOutputCommitter(); committer.setupJob(jobContext); but it was deleted from patch-5785-2.txt, thus _temporary won't be created if the variable is set. Can anybody tell me why ?
          Hide
          yuling added a comment -

          _temporary is create at clearnup phase,and used at some OutputFormat, when mapred.committer.job.setup.cleanup.needed is set to false, we will have to create _temporary before we use it.

          Show
          yuling added a comment - _temporary is create at clearnup phase,and used at some OutputFormat, when mapred.committer.job.setup.cleanup.needed is set to false, we will have to create _temporary before we use it.

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              Owen O'Malley
            • Votes:
              3 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development