Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1476

committer.needsTaskCommit should not be called for a task cleanup attempt

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed Map/Reduce framework to not call commit task for special tasks like job setup/cleanup and task cleanup.

      Description

      Currently, Task.done() calls committer.needsTaskCommit() to know whether it needs a commit or not. This need not be called for task cleanup attempt as no commit is required for a cleanup attempt.
      Due to MAPREDUCE-1409, we saw a case where cleanup attempt went into COMMIT_PENDING state.

      1. mr-1476-y20.patch
        10 kB
        Hemanth Yamijala
      2. patch-1476.txt
        7 kB
        Amareshwari Sriramadasu
      3. patch-1476-1.txt
        9 kB
        Hemanth Yamijala
      4. patch-1476-2.txt
        11 kB
        Hemanth Yamijala
      5. patch-1476-ydist.txt
        10 kB
        Amareshwari Sriramadasu

        Activity

        Hide
        Amareshwari Sriramadasu added a comment -

        Patch fixing the bug. Added a unit test which fails without the patch and passes with the patch.

        Show
        Amareshwari Sriramadasu added a comment - Patch fixing the bug. Added a unit test which fails without the patch and passes with the patch.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12435549/patch-1476.txt
        against trunk revision 908321.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/316/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/316/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/316/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/316/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12435549/patch-1476.txt against trunk revision 908321. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/316/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/316/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/316/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/316/console This message is automatically generated.
        Hide
        Hemanth Yamijala added a comment -

        So, just to be sure - the patch is going to attempt commit of a task only for true Map/Reduce tasks. Job setup/cleanup and task cleanup all cannot rely on a commit action at all. As this is a change in semantics of tasks, I wanted to call this out specifically, though I feel this is fine. Any objections to this from anyone ?

        Show
        Hemanth Yamijala added a comment - So, just to be sure - the patch is going to attempt commit of a task only for true Map/Reduce tasks. Job setup/cleanup and task cleanup all cannot rely on a commit action at all. As this is a change in semantics of tasks, I wanted to call this out specifically, though I feel this is fine. Any objections to this from anyone ?
        Hide
        Hemanth Yamijala added a comment -

        Code changes look good. Unfortunately, I did not understand the test case fully to comment on it. I did notice though that we are testing the scenario only for task cleanup, while I suppose we do need to test for job setup and cleanup as well. I wonder if a more thorough unit test would be enabled if we refactor the logic of deciding if commit is required into a separate method and directly test that method with all types of tasks. I created such a method and some tests in the attached patch. Does this make sense ?

        Show
        Hemanth Yamijala added a comment - Code changes look good. Unfortunately, I did not understand the test case fully to comment on it. I did notice though that we are testing the scenario only for task cleanup, while I suppose we do need to test for job setup and cleanup as well. I wonder if a more thorough unit test would be enabled if we refactor the logic of deciding if commit is required into a separate method and directly test that method with all types of tasks. I created such a method and some tests in the attached patch. Does this make sense ?
        Hide
        Hemanth Yamijala added a comment -

        Amarsri and I discussed the test case in the original patch and decided to retain it as a regression test for the bug that triggered this JIRA, in addition to more targeted unit tests that I wrote.

        The new patch has the following changes, in line with the above decision:

        • Makes editorial changes to the regression test - adding a few more comments, refactoring for more explanatory names, etc.
        • Uses a different committer for the targeted unit tests that always return true for needTaskCommit.
        • Fixes TestTaskCleanup.tearDown to call super.

        Amarsri, can you please take a quick look at the changes and run through Hudson ?

        Show
        Hemanth Yamijala added a comment - Amarsri and I discussed the test case in the original patch and decided to retain it as a regression test for the bug that triggered this JIRA, in addition to more targeted unit tests that I wrote. The new patch has the following changes, in line with the above decision: Makes editorial changes to the regression test - adding a few more comments, refactoring for more explanatory names, etc. Uses a different committer for the targeted unit tests that always return true for needTaskCommit. Fixes TestTaskCleanup.tearDown to call super. Amarsri, can you please take a quick look at the changes and run through Hudson ?
        Hide
        Amareshwari Sriramadasu added a comment -

        Changes look fine.
        Submitting for hudson

        Show
        Amareshwari Sriramadasu added a comment - Changes look fine. Submitting for hudson
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch for Yahoo! distribution.
        Ran ant test and test-patch. All tests passed except TestHdfsProxy.

        Show
        Amareshwari Sriramadasu added a comment - Patch for Yahoo! distribution. Ran ant test and test-patch. All tests passed except TestHdfsProxy.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12435846/patch-1476-2.txt
        against trunk revision 909993.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/454/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/454/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/454/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/454/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12435846/patch-1476-2.txt against trunk revision 909993. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/454/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/454/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/454/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/454/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        -1 core tests.

        TestCommandLineJobSubmission failed with NoClassDefFoundError. See MAPREDUCE-1275.
        Stacktrace for the failure:

        java.lang.NoClassDefFoundError: org/apache/hadoop/ipc/Server$Handler
        	at org.apache.hadoop.ipc.Server.start(Server.java:1343)
        	at org.apache.hadoop.hdfs.server.namenode.NameNode.activate(NameNode.java:318)
        	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:308)
        	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421)
        	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:410)
        	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1261)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:278)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:127)
        
        Show
        Amareshwari Sriramadasu added a comment - -1 core tests. TestCommandLineJobSubmission failed with NoClassDefFoundError. See MAPREDUCE-1275 . Stacktrace for the failure: java.lang.NoClassDefFoundError: org/apache/hadoop/ipc/Server$Handler at org.apache.hadoop.ipc.Server.start(Server.java:1343) at org.apache.hadoop.hdfs.server.namenode.NameNode.activate(NameNode.java:318) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:308) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:421) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:410) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1261) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:278) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:127)
        Hide
        Hemanth Yamijala added a comment -

        I just committed this to trunk. Thanks, Amareshwari !

        Show
        Hemanth Yamijala added a comment - I just committed this to trunk. Thanks, Amareshwari !
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #241 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/241/)
        . Fix the M/R framework to not call commit for special tasks like job setup/cleanup and task cleanup. Contributed by Amareshwari Sriramadasu.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #241 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/241/ ) . Fix the M/R framework to not call commit for special tasks like job setup/cleanup and task cleanup. Contributed by Amareshwari Sriramadasu.
        Hide
        Hemanth Yamijala added a comment -

        An updated patch for earlier version of hadoop. Not for commit here.

        Show
        Hemanth Yamijala added a comment - An updated patch for earlier version of hadoop. Not for commit here.

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Amareshwari Sriramadasu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development