Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1302

TrackerDistributedCacheManager can delete file asynchronously

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.2, 0.21.0, 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      With the help of AsyncDiskService from MAPREDUCE-1213, we should be able to delete files from distributed cache asynchronously.

      That will help make task initialization faster, because task initialization calls the code that localizes files into the cache and may delete some other files.
      The deletion can slow down the task initialization speed.

      1. MAPREDUCE-1302.branch-0.20.on.top.of.MAPREDUCE-1213.3.patch
        42 kB
        Zheng Shao
      2. MAPREDUCE-1302.branch-0.20.on.top.of.MAPREDUCE-1213.2.patch
        42 kB
        Zheng Shao
      3. MAPREDUCE-1302.5.patch
        26 kB
        Zheng Shao
      4. MAPREDUCE-1302.4.patch
        26 kB
        Zheng Shao
      5. MAPREDUCE-1302.3.patch
        22 kB
        Zheng Shao
      6. MAPREDUCE-1302.2.patch
        21 kB
        Zheng Shao
      7. MAPREDUCE-1302.1.patch
        8 kB
        Zheng Shao
      8. MAPREDUCE-1302.0.patch
        19 kB
        Zheng Shao

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          26d 18h 41m 6 Zheng Shao 12/Jan/10 02:27
          Open Open Patch Available Patch Available
          8m 41s 7 Zheng Shao 12/Jan/10 02:27
          Patch Available Patch Available Resolved Resolved
          17h 24m 1 dhruba borthakur 12/Jan/10 19:52
          Resolved Resolved Closed Closed
          224d 1h 27m 1 Tom White 24/Aug/10 22:19
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12314045 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #206 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/206/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #206 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/206/ )
          Zheng Shao made changes -
          Hide
          Zheng Shao added a comment -

          Fixed some unit test failures for hadoop-0.20. Note that this patch can only be applied to hadoop-0.20 after the MAPREDUCE-1213 is applied.

          Show
          Zheng Shao added a comment - Fixed some unit test failures for hadoop-0.20. Note that this patch can only be applied to hadoop-0.20 after the MAPREDUCE-1213 is applied.
          Zheng Shao made changes -
          Attachment MAPREDUCE-1302.branch-0.20.on.top.of.MAPREDUCE-1213.patch [ 12430070 ]
          Zheng Shao made changes -
          Zheng Shao made changes -
          Attachment MAPREDUCE-1302.branch-0.20.on.top.of.MAPREDUCE-1213.patch [ 12430070 ]
          Hide
          Zheng Shao added a comment -

          This patch is for branch-0.20 (on top of MAPREDUCE-1213.branch-0.20.patch)

          Show
          Zheng Shao added a comment - This patch is for branch-0.20 (on top of MAPREDUCE-1213 .branch-0.20.patch)
          dhruba borthakur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.22.0 [ 12314184 ]
          Resolution Fixed [ 1 ]
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Zheng!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Zheng!
          Hide
          Scott Chen added a comment -

          Have run TestTrackerDistributedCacheManagerWithLinuxTaskController. It passed.

          Testcase: testManagerFlow took 0.015 sec
          Testcase: testReferenceCount took 0.002 sec
          Testcase: testPublicPrivateCache took 0.004 sec
          Testcase: testDeleteCache took 0.002 sec
          Testcase: testFileSystemOtherThanDefault took 0.002 sec
          Testcase: testFreshness took 0.003 sec
          Testcase: testCustomPermissions took 0.002 sec

          Show
          Scott Chen added a comment - Have run TestTrackerDistributedCacheManagerWithLinuxTaskController. It passed. Testcase: testManagerFlow took 0.015 sec Testcase: testReferenceCount took 0.002 sec Testcase: testPublicPrivateCache took 0.004 sec Testcase: testDeleteCache took 0.002 sec Testcase: testFileSystemOtherThanDefault took 0.002 sec Testcase: testFreshness took 0.003 sec Testcase: testCustomPermissions took 0.002 sec
          Hide
          Vinod Kumar Vavilapalli added a comment -

          +1 for the latest patch.

          Show
          Vinod Kumar Vavilapalli added a comment - +1 for the latest patch.
          Hide
          Amareshwari Sriramadasu added a comment -

          Since Hudson does not run LinuxTaskController tests, can you run TestTrackerDistributedCacheManagerWithLinuxTaskController and make sure it passes?

          Show
          Amareshwari Sriramadasu added a comment - Since Hudson does not run LinuxTaskController tests, can you run TestTrackerDistributedCacheManagerWithLinuxTaskController and make sure it passes?
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12429929/MAPREDUCE-1302.5.patch
          against trunk revision 898019.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/375/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/375/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/375/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/375/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429929/MAPREDUCE-1302.5.patch against trunk revision 898019. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/375/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/375/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/375/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/375/console This message is automatically generated.
          Zheng Shao made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Zheng Shao made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Zheng Shao made changes -
          Attachment MAPREDUCE-1302.5.patch [ 12429929 ]
          Hide
          Zheng Shao added a comment -

          Merged with latest trunk.
          Vinod, can you take a look?

          Show
          Zheng Shao added a comment - Merged with latest trunk. Vinod, can you take a look?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Zheng, sorry for the delay, got caught up with other things. MAPREDUCE-1186 got committed which invalidated your latest patch. It changed the constructor of MRAsyncDiskService and Tasktracker.initialize(). Can you please generate a new one?

          I myself manually merged the changes for review of code. Rest of the changes look good. I'll do a +1 once you upload the updated patch. Thanks for being patient!

          Show
          Vinod Kumar Vavilapalli added a comment - Zheng, sorry for the delay, got caught up with other things. MAPREDUCE-1186 got committed which invalidated your latest patch. It changed the constructor of MRAsyncDiskService and Tasktracker.initialize(). Can you please generate a new one? I myself manually merged the changes for review of code. Rest of the changes look good. I'll do a +1 once you upload the updated patch. Thanks for being patient!
          Hide
          Zheng Shao added a comment -

          Scott, I do use the returned value of #getRelativePathName() after comparing it with null.
          So the other option is to have 2 functions: #isInVolume() and #getRelativePathName().
          I prefer having only 1 function just for simplification.

          Show
          Zheng Shao added a comment - Scott, I do use the returned value of #getRelativePathName() after comparing it with null. So the other option is to have 2 functions: #isInVolume() and #getRelativePathName(). I prefer having only 1 function just for simplification.
          Hide
          Scott Chen added a comment -

          +1 to the final patch.
          I only have a minor comment. Can we change #getRelativePathName() to #isInVolume() instead? Just return true if the absoluteFileName is in the volume. It is actually how this function is used right now. This is not necessary. Just to make it more clear.

          Show
          Scott Chen added a comment - +1 to the final patch. I only have a minor comment. Can we change #getRelativePathName() to #isInVolume() instead? Just return true if the absoluteFileName is in the volume. It is actually how this function is used right now. This is not necessary. Just to make it more clear.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12429575/MAPREDUCE-1302.4.patch
          against trunk revision 896265.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/250/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/250/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/250/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/250/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429575/MAPREDUCE-1302.4.patch against trunk revision 896265. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/250/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/250/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/250/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/250/console This message is automatically generated.
          Zheng Shao made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Zheng Shao made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Zheng Shao made changes -
          Attachment MAPREDUCE-1302.4.patch [ 12429575 ]
          Hide
          Zheng Shao added a comment -

          Modified according to Vinod's comments.

          I didn't change the test. I did verify the deletion in the makeSureCleanedUp() method.

          I will deprecate JobConf.deleteLocalFiles(subdir) in a follow-up jira.

          Show
          Zheng Shao added a comment - Modified according to Vinod's comments. I didn't change the test. I did verify the deletion in the makeSureCleanedUp() method. I will deprecate JobConf.deleteLocalFiles(subdir) in a follow-up jira.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I've looked at your latest patch. Some comments I have on the patch:

          TaskTracker.java

          • #getMRAsyncDiskService() method is not used anywhere. Can we remove it?
          • We should add javadoc of #cleanupStorage() method as to what one should use instead of this deprecated method.
          • The method calls this.distributedCacheManager.purgeCache() and asyncDiskService.moveAndDeleteAllVolumes() (at +645-46) are not needed at all because there is a asyncDiskService.moveAndDeleteFromEachVolume(SUBDIR) (at +575) itself that completely clears the taskTracker subdir. They can be removed.

          MRAsyncDiskService.java

          • There is an unused import of Configuration which can be removed.
          • Annotate MRAsyncDiskService as @InterfaceAudience.Private as it is for internal use only.
          • Please change the java comment of the class to javadoc. Also we should make it clear that clients of this service should NOT write files themselves in the TOBEDELTED directory in any of the volumes or risk losing such files.
          • Documentation (javadoc) is needed as to how MRAsyncDiskService(JobConf conf) uses the 'conf' parameter.
          • Can we rename moveAndDeleteAllVolumes() to something like cleanupAllVolumes() or similar. This is because this method is only moving and deleting the contents of the volumes and not the volumes themselves.
          • To be more clear, can we also rename moveAndDelete(String absolutePathName) to moveAndDeleteAbsolutePath(String absolutePathName) and moveAndDelete(String volume, String pathName) to moveAndDeleteRelativePath(String volume, String pathName)?
          • In any case, we should make the absolute/relative difference explicit in the javadoc of both the methods.
          • getRelativePathName() is bit buggy in that it will return null if any one or both of 'volume' string and 'absolutePathName' string have multiple Path.SEPARATOR characters in them. We should construct normalized paths before working on them.
          • Similar to above, we should make sure that paths/path fragments are normalized before their comparision in moveAndDeleteAllVolumes() at +289 and before passing to DeleteTask at +246 moveAndDelete(String volume, String pathName)
          • Javadoc of moveAndDeleteAllVolumes() needs to be fixed, it looks incomplete now
          • Nit: Need to fix the comment at +288 to use TOBEDELETED instead of SUBDIR
          • Not related to this JIRA: I think that uniqueId generation should be static and be valid across different MRAsyncDiskService objects. We can file another issue to fix this if you agree that it is a bug here.

          JobConf

          • When deprecating Jobconf.deleteLocalFiles(), we should also modify javadoc to state that MRAsyncDiskService should be used instead.
          • What about JobConf.deleteLocalFiles(String subDir) and its callers? Another JIRA?

          Testing:

          • Even though the inline removal of files is tested by various other test-cases, we haven't test the actual deletion anywhere. For this, we can refactor the DeleteTask part of moveAndDelete(String, String) into a new method and can then override it with inline deletion to assert the actual deletion of files by verifying the contents of TOBEDELETED directory.
          Show
          Vinod Kumar Vavilapalli added a comment - I've looked at your latest patch. Some comments I have on the patch: TaskTracker.java #getMRAsyncDiskService() method is not used anywhere. Can we remove it? We should add javadoc of #cleanupStorage() method as to what one should use instead of this deprecated method. The method calls this.distributedCacheManager.purgeCache() and asyncDiskService.moveAndDeleteAllVolumes() (at +645-46) are not needed at all because there is a asyncDiskService.moveAndDeleteFromEachVolume(SUBDIR) (at +575) itself that completely clears the taskTracker subdir. They can be removed. MRAsyncDiskService.java There is an unused import of Configuration which can be removed. Annotate MRAsyncDiskService as @InterfaceAudience.Private as it is for internal use only. Please change the java comment of the class to javadoc. Also we should make it clear that clients of this service should NOT write files themselves in the TOBEDELTED directory in any of the volumes or risk losing such files. Documentation (javadoc) is needed as to how MRAsyncDiskService(JobConf conf) uses the 'conf' parameter. Can we rename moveAndDeleteAllVolumes() to something like cleanupAllVolumes() or similar. This is because this method is only moving and deleting the contents of the volumes and not the volumes themselves. To be more clear, can we also rename moveAndDelete(String absolutePathName) to moveAndDeleteAbsolutePath(String absolutePathName) and moveAndDelete(String volume, String pathName) to moveAndDeleteRelativePath(String volume, String pathName) ? In any case, we should make the absolute/relative difference explicit in the javadoc of both the methods. getRelativePathName() is bit buggy in that it will return null if any one or both of 'volume' string and 'absolutePathName' string have multiple Path.SEPARATOR characters in them. We should construct normalized paths before working on them. Similar to above, we should make sure that paths/path fragments are normalized before their comparision in moveAndDeleteAllVolumes() at +289 and before passing to DeleteTask at +246 moveAndDelete(String volume, String pathName) Javadoc of moveAndDeleteAllVolumes() needs to be fixed, it looks incomplete now Nit: Need to fix the comment at +288 to use TOBEDELETED instead of SUBDIR Not related to this JIRA: I think that uniqueId generation should be static and be valid across different MRAsyncDiskService objects. We can file another issue to fix this if you agree that it is a bug here. JobConf When deprecating Jobconf.deleteLocalFiles(), we should also modify javadoc to state that MRAsyncDiskService should be used instead. What about JobConf.deleteLocalFiles(String subDir) and its callers? Another JIRA? Testing: Even though the inline removal of files is tested by various other test-cases, we haven't test the actual deletion anywhere. For this, we can refactor the DeleteTask part of moveAndDelete(String, String) into a new method and can then override it with inline deletion to assert the actual deletion of files by verifying the contents of TOBEDELETED directory.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12429498/MAPREDUCE-1302.3.patch
          against trunk revision 896265.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/358/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/358/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/358/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/358/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429498/MAPREDUCE-1302.3.patch against trunk revision 896265. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/358/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/358/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/358/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/358/console This message is automatically generated.
          Zheng Shao made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Zheng Shao made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Zheng Shao made changes -
          Attachment MAPREDUCE-1302.3.patch [ 12429498 ]
          Hide
          Zheng Shao added a comment -

          Renamed SUBDIR to TOBEDELETED to avoid confusion.

          Show
          Zheng Shao added a comment - Renamed SUBDIR to TOBEDELETED to avoid confusion.
          Hide
          Zheng Shao added a comment -

          MAPREDUCE-1141 is fixed by this patch.

          Show
          Zheng Shao added a comment - MAPREDUCE-1141 is fixed by this patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12429084/MAPREDUCE-1302.2.patch
          against trunk revision 894165.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/350/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/350/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/350/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/350/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429084/MAPREDUCE-1302.2.patch against trunk revision 894165. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/350/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/350/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/350/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/350/console This message is automatically generated.
          Zheng Shao made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Zheng Shao made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Zheng Shao added a comment -

          The code didn't create a single task for toBeDeleted, but went through the toBeDeleted directory and create one task per each.
          The reason for that is:
          1. This allows parallel deletion of the contents inside toBeDeleted
          2. A single list call per volume shouldn't take too long
          3. If we want to create a single task for toBeDeleted, then we need to rename it to something else, and recreate toBeDeleted, and then move the old one to be a sub directory inside the new toBeDeleted. This will introduce additional intermediate states that may be hard to recover from.

          Show
          Zheng Shao added a comment - The code didn't create a single task for toBeDeleted, but went through the toBeDeleted directory and create one task per each. The reason for that is: 1. This allows parallel deletion of the contents inside toBeDeleted 2. A single list call per volume shouldn't take too long 3. If we want to create a single task for toBeDeleted, then we need to rename it to something else, and recreate toBeDeleted, and then move the old one to be a sub directory inside the new toBeDeleted. This will introduce additional intermediate states that may be hard to recover from.
          Zheng Shao made changes -
          Attachment MAPREDUCE-1302.2.patch [ 12429084 ]
          Hide
          Zheng Shao added a comment -

          Added logic to remove the files inside toBeDeleted upon restart.

          Show
          Zheng Shao added a comment - Added logic to remove the files inside toBeDeleted upon restart.
          Hide
          Zheng Shao added a comment -

          Good question. There is no special handling right now.
          I will list the directory and create one task per item returned.

          Show
          Zheng Shao added a comment - Good question. There is no special handling right now. I will list the directory and create one task per item returned.
          Hide
          dhruba borthakur added a comment -

          The code looks good to me. What happens to a file if the TaskTracker dies after renaming the file but before the async thread gets around to deleting the file?

          Show
          dhruba borthakur added a comment - The code looks good to me. What happens to a file if the TaskTracker dies after renaming the file but before the async thread gets around to deleting the file?
          Hide
          Zheng Shao added a comment -

          The test errors are not related.

          The first one seems like random error - it didn't appear on the last hudson test of the same patch.
          The next 3 are common to all patches' test results.

          >>><<< org.apache.hadoop.security.authorize.TestServiceLevelAuthorization.testServiceLevelAuthorization
          >>><<< org.apache.hadoop.streaming.TestStreamingExitStatus.testMapFailOk
          >>><<< org.apache.hadoop.streaming.TestStreamingExitStatus.testReduceFailOk
          >>><<< org.apache.hadoop.streaming.TestStreamingKeyValue.testCommandLine

          Show
          Zheng Shao added a comment - The test errors are not related. The first one seems like random error - it didn't appear on the last hudson test of the same patch. The next 3 are common to all patches' test results. >>><<< org.apache.hadoop.security.authorize.TestServiceLevelAuthorization.testServiceLevelAuthorization >>><<< org.apache.hadoop.streaming.TestStreamingExitStatus.testMapFailOk >>><<< org.apache.hadoop.streaming.TestStreamingExitStatus.testReduceFailOk >>><<< org.apache.hadoop.streaming.TestStreamingKeyValue.testCommandLine
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12428349/MAPREDUCE-1302.1.patch
          against trunk revision 892479.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/227/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/227/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/227/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/227/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428349/MAPREDUCE-1302.1.patch against trunk revision 892479. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/227/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/227/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/227/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/227/console This message is automatically generated.
          Zheng Shao made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Zheng Shao added a comment -

          Transient errors in hudson. (user1 not found)
          Submitting again.

          Show
          Zheng Shao added a comment - Transient errors in hudson. (user1 not found) Submitting again.
          Zheng Shao made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12428349/MAPREDUCE-1302.1.patch
          against trunk revision 891920.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/215/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/215/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/215/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/215/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428349/MAPREDUCE-1302.1.patch against trunk revision 891920. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/215/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/215/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/215/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/215/console This message is automatically generated.
          Zheng Shao made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Zheng Shao made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Zheng Shao made changes -
          Attachment MAPREDUCE-1302.1.patch [ 12428349 ]
          Hide
          Zheng Shao added a comment -

          This patch is on top of MAPREDUCE-1213 which is already committed.

          Show
          Zheng Shao added a comment - This patch is on top of MAPREDUCE-1213 which is already committed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12428138/MAPREDUCE-1302.0.patch
          against trunk revision 891111.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/201/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/201/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/201/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/201/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428138/MAPREDUCE-1302.0.patch against trunk revision 891111. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/201/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/201/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/201/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/201/console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Linking the blocking JIRA issue for the sake of MRAsyncDiskService.

          Show
          Vinod Kumar Vavilapalli added a comment - Linking the blocking JIRA issue for the sake of MRAsyncDiskService .
          Vinod Kumar Vavilapalli made changes -
          Link This issue is blocked by MAPREDUCE-1213 [ MAPREDUCE-1213 ]
          Vinod Kumar Vavilapalli made changes -
          Affects Version/s 0.20.2 [ 12314205 ]
          Affects Version/s 0.21.0 [ 12314045 ]
          Affects Version/s 0.22.0 [ 12314184 ]
          Component/s tasktracker [ 12312906 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Not doing it asynchronously is actually a bug, see MAPREDUCE-1141 which is resolved as duplicate of this one.

          Show
          Vinod Kumar Vavilapalli added a comment - Not doing it asynchronously is actually a bug, see MAPREDUCE-1141 which is resolved as duplicate of this one.
          Hide
          Ravi Gummadi added a comment -

          deletion of contents of work dir in TaskTracker.SetupWorkDir() should also be done asynchronously(as in MAPREDUCE-1213, by first moving to toBeDeleted and then removing by a separate thread(could be CleanupQueue)) sothat the next task in the same jvm(in case of jvm reuse) can start immediately. Currently this deletion is done inline, which can take a long time based on the number of files existing in work dir.

          Show
          Ravi Gummadi added a comment - deletion of contents of work dir in TaskTracker.SetupWorkDir() should also be done asynchronously(as in MAPREDUCE-1213 , by first moving to toBeDeleted and then removing by a separate thread(could be CleanupQueue)) sothat the next task in the same jvm(in case of jvm reuse) can start immediately. Currently this deletion is done inline, which can take a long time based on the number of files existing in work dir.
          Vinod Kumar Vavilapalli made changes -
          Link This issue duplicates MAPREDUCE-1141 [ MAPREDUCE-1141 ]
          Hide
          Amareshwari Sriramadasu added a comment -

          There is known issue with deletion, see MAPREDUCE-1141. Deletion should not happen by task localization, it should be done by separate thread in TaskTracker (CleanupQueue ?).

          Show
          Amareshwari Sriramadasu added a comment - There is known issue with deletion, see MAPREDUCE-1141 . Deletion should not happen by task localization, it should be done by separate thread in TaskTracker (CleanupQueue ?).
          Zheng Shao made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Zheng Shao made changes -
          Field Original Value New Value
          Attachment MAPREDUCE-1302.0.patch [ 12428138 ]
          Hide
          Zheng Shao added a comment -

          This patch includes MAPREDUCE-1213. It's just for demo purpose.

          Show
          Zheng Shao added a comment - This patch includes MAPREDUCE-1213 . It's just for demo purpose.
          Zheng Shao created issue -

            People

            • Assignee:
              Zheng Shao
              Reporter:
              Zheng Shao
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development