Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3067

Container exit status not set properly to launched process's exit code on successful completion of process

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None
    • Tags:
      mr2,mapreduce-2.0

      Description

      When testing the distributed shell sample app master, the container exit status was being returned incorrectly.

      11/09/21 11:32:58 INFO DistributedShell.ApplicationMaster: Got container status for containerID= container_1316629955324_0001_01_000002, state=COMPLETE, exitStatus=-1000, diagnostics=

      1. MR-3067.2.patch
        7 kB
        Hitesh Shah
      2. MR-3067.1.patch
        7 kB
        Hitesh Shah

        Activity

        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #843 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/843/)
        MAPREDUCE-3067. Ensure exit-code is set correctly for containers. Contributed by Hitesh Shah.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176235
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #843 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/843/ ) MAPREDUCE-3067 . Ensure exit-code is set correctly for containers. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176235 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #813 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/813/)
        MAPREDUCE-3067. Ensure exit-code is set correctly for containers. Contributed by Hitesh Shah.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176235
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #813 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/813/ ) MAPREDUCE-3067 . Ensure exit-code is set correctly for containers. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176235 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #29 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/29/)
        Merge -r 1176234:1176235 from trunk to branch-0.23 to fix MAPREDUCE-3067.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176236
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #29 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/29/ ) Merge -r 1176234:1176235 from trunk to branch-0.23 to fix MAPREDUCE-3067 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176236 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #22 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/22/)
        Merge -r 1176234:1176235 from trunk to branch-0.23 to fix MAPREDUCE-3067.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176236
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #22 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/22/ ) Merge -r 1176234:1176235 from trunk to branch-0.23 to fix MAPREDUCE-3067 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176236 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #980 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/980/)
        MAPREDUCE-3067. Ensure exit-code is set correctly for containers. Contributed by Hitesh Shah.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176235
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #980 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/980/ ) MAPREDUCE-3067 . Ensure exit-code is set correctly for containers. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176235 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1038 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1038/)
        MAPREDUCE-3067. Ensure exit-code is set correctly for containers. Contributed by Hitesh Shah.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176235
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1038 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1038/ ) MAPREDUCE-3067 . Ensure exit-code is set correctly for containers. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176235 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #960 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/960/)
        MAPREDUCE-3067. Ensure exit-code is set correctly for containers. Contributed by Hitesh Shah.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176235
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #960 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/960/ ) MAPREDUCE-3067 . Ensure exit-code is set correctly for containers. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176235 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Arun C Murthy added a comment -

        +1, lgtm.

        I just committed this. Thanks Hitesh!

        Show
        Arun C Murthy added a comment - +1, lgtm. I just committed this. Thanks Hitesh!
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12496578/MR-3067.2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/861//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/861//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12496578/MR-3067.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/861//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/861//console This message is automatically generated.
        Hitesh Shah made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hitesh Shah made changes -
        Attachment MR-3067.2.patch [ 12496578 ]
        Hide
        Hitesh Shah added a comment -

        Fixed error in LCE tests.

        Show
        Hitesh Shah added a comment - Fixed error in LCE tests.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12496571/MR-3067.1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/860//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/860//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12496571/MR-3067.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/860//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/860//console This message is automatically generated.
        Hitesh Shah made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Hitesh Shah added a comment -

        Minor copy-paste fix pending

        Show
        Hitesh Shah added a comment - Minor copy-paste fix pending
        Hitesh Shah made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hitesh Shah made changes -
        Attachment MR-3067.1.patch [ 12496571 ]
        Hide
        Hitesh Shah added a comment -

        Trivial fix with modified unit tests.

        Show
        Hitesh Shah added a comment - Trivial fix with modified unit tests.
        Arun C Murthy made changes -
        Priority Major [ 3 ] Blocker [ 1 ]
        Hide
        Vinod Kumar Vavilapalli added a comment -

        [..] why the exit status does not need to be checked for map/reduce task containers.

        In MRV2 framework, successful maps and reduces directly talk to the AM via TaskUmbilicalProtocol.done(). If the task JVM happens to crash after the done() call, it doesn't really affect the real outcome of the task - so no point in checking the exit code to be zero. For failed and crashed tasks, we can just use the exit-code to let the user know via web-UI/command-line.

        Other frameworks, like the DistributedShell you are writing, may not have this direct communication between container and AM and so can leverage the exit-code for figuring out the outcome of the containers.

        Show
        Vinod Kumar Vavilapalli added a comment - [..] why the exit status does not need to be checked for map/reduce task containers. In MRV2 framework, successful maps and reduces directly talk to the AM via TaskUmbilicalProtocol.done() . If the task JVM happens to crash after the done() call, it doesn't really affect the real outcome of the task - so no point in checking the exit code to be zero. For failed and crashed tasks, we can just use the exit-code to let the user know via web-UI/command-line. Other frameworks, like the DistributedShell you are writing, may not have this direct communication between container and AM and so can leverage the exit-code for figuring out the outcome of the containers.
        Hide
        Hitesh Shah added a comment -

        Ate a couple of words in that statement. The code in RMContainerAllocator currently keeps a count of completed maps and reduces but does not seem to check the exit status. For the sake of documentation, it would be good if you could clarify as to why the exit status does not need to be checked for map/reduce task containers.

        Show
        Hitesh Shah added a comment - Ate a couple of words in that statement. The code in RMContainerAllocator currently keeps a count of completed maps and reduces but does not seem to check the exit status. For the sake of documentation, it would be good if you could clarify as to why the exit status does not need to be checked for map/reduce task containers.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Good catch, Hitesh!

        Second aspect to this is the exit status is checked on completion of map or reduce tasks.

        This isn't clear to me, more details please?

        Show
        Vinod Kumar Vavilapalli added a comment - Good catch, Hitesh! Second aspect to this is the exit status is checked on completion of map or reduce tasks. This isn't clear to me, more details please?
        Arun C Murthy made changes -
        Field Original Value New Value
        Assignee Hitesh Shah [ hitesh ]
        Hide
        Hitesh Shah added a comment -

        Second aspect to this is the exit status is checked on completion of map or reduce tasks.

        Show
        Hitesh Shah added a comment - Second aspect to this is the exit status is checked on completion of map or reduce tasks.
        Hide
        Hitesh Shah added a comment -

        Possible patch for addressing part of the issue.

        — a/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
        +++ b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
        @@ -554,6 +554,9 @@ public class ContainerImpl implements Container {
        static class ExitedWithSuccessTransition extends ContainerTransition {
        @Override
        public void transition(ContainerImpl container, ContainerEvent event) {
        + // Set exit code to 0 to denote success
        + container.exitCode = 0;
        +
        // TODO: Add containerWorkDir to the deletion service.

        // Inform the localizer to decrement reference counts and cleanup

        Show
        Hitesh Shah added a comment - Possible patch for addressing part of the issue. — a/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java +++ b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java @@ -554,6 +554,9 @@ public class ContainerImpl implements Container { static class ExitedWithSuccessTransition extends ContainerTransition { @Override public void transition(ContainerImpl container, ContainerEvent event) { + // Set exit code to 0 to denote success + container.exitCode = 0; + // TODO: Add containerWorkDir to the deletion service. // Inform the localizer to decrement reference counts and cleanup
        Hitesh Shah created issue -

          People

          • Assignee:
            Hitesh Shah
            Reporter:
            Hitesh Shah
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development