Hadoop Common
  1. Hadoop Common
  2. HADOOP-10312

Shell.ExitCodeException to have more useful toString

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.5.0
    • Component/s: util
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Shell's ExitCodeException doesn't include the exit code in the toString value, so isn't that useful in diagnosing container start failures in YARN

      1. HADOOP-10312-001.patch
        1 kB
        Steve Loughran
      2. HADOOP-10312-002.patch
        1 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          Steve Loughran added a comment -

          Example

          2014-01-30 14:20:12,042 [AMRM Callback Handler Thread] INFO  HoyaAppMaster.yarn (HoyaAppMaster.java:onContainersCompleted(839)) - Container Completion for containerID=container_1391075472386_0004_01_000032, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: 
          org.apache.hadoop.util.Shell$ExitCodeException: 
          	at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
          	at org.apache.hadoop.util.Shell.run(Shell.java:418)
          	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
          	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
          	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
          	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          	at java.lang.Thread.run(Thread.java:744)
          

          it failed, but it's not obvious why

          Show
          Steve Loughran added a comment - Example 2014-01-30 14:20:12,042 [AMRM Callback Handler Thread ] INFO HoyaAppMaster.yarn (HoyaAppMaster.java:onContainersCompleted(839)) - Container Completion for containerID=container_1391075472386_0004_01_000032, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:505) at org.apache.hadoop.util.Shell.run(Shell.java:418) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:744) it failed, but it's not obvious why
          Hide
          Chris Nauroth added a comment -

          I've always wanted it to be easier to capture stdout/stderr from failed external commands too. It could be potentially spammy though, so maybe it would need to go behind debug log level somehow.

          If you want to keep the scope of this issue to just the exit code, please go ahead, and I'll file a separate issue related to capturing stdout/stderr.

          Show
          Chris Nauroth added a comment - I've always wanted it to be easier to capture stdout/stderr from failed external commands too. It could be potentially spammy though, so maybe it would need to go behind debug log level somehow. If you want to keep the scope of this issue to just the exit code, please go ahead, and I'll file a separate issue related to capturing stdout/stderr.
          Hide
          Steve Loughran added a comment -

          adds a toString operator with the exit code and marks the exit code as final.

          The exit code field is still left as protected -there's no direct accessors and we could make it private

          Show
          Steve Loughran added a comment - adds a toString operator with the exit code and marks the exit code as final. The exit code field is still left as protected -there's no direct accessors and we could make it private
          Hide
          Chris Nauroth added a comment -

          I'm in favor of making exitCode private. I looked back on HADOOP-2512, which introduced this exception class. There doesn't appear to be any historical reason that it needs to be package-private.

          Show
          Chris Nauroth added a comment - I'm in favor of making exitCode private. I looked back on HADOOP-2512 , which introduced this exception class. There doesn't appear to be any historical reason that it needs to be package-private.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12651542/HADOOP-10312-001.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4110//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4110//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12651542/HADOOP-10312-001.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4110//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4110//console This message is automatically generated.
          Hide
          Chris Nauroth added a comment -

          -1 tests included. The patch doesn't appear to include any new or modified tests.

          This patch doesn't require a test, IMO.

          Show
          Chris Nauroth added a comment - -1 tests included. The patch doesn't appear to include any new or modified tests. This patch doesn't require a test, IMO.
          Hide
          Steve Loughran added a comment -

          ...exit code field is now private

          Show
          Steve Loughran added a comment - ...exit code field is now private
          Hide
          Chris Nauroth added a comment -

          +1 for the patch, pending Jenkins run. Thanks, Steve!

          Show
          Chris Nauroth added a comment - +1 for the patch, pending Jenkins run. Thanks, Steve!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12651557/HADOOP-10312-002.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4116//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4116//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12651557/HADOOP-10312-002.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4116//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4116//console This message is automatically generated.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5817 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5817/)
          HADOOP-10312 Shell.ExitCodeException to have more useful toString (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607591)

          • /hadoop/common/trunk
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5817 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5817/ ) HADOOP-10312 Shell.ExitCodeException to have more useful toString (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607591 ) /hadoop/common/trunk /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5820 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5820/)
          HADOOP-10312 changes.text updated in wrong place (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607631)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5820 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5820/ ) HADOOP-10312 changes.text updated in wrong place (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607631 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1793 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1793/)
          HADOOP-10312 Shell.ExitCodeException to have more useful toString (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607591)

          • /hadoop/common/trunk
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1793 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1793/ ) HADOOP-10312 Shell.ExitCodeException to have more useful toString (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607591 ) /hadoop/common/trunk /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1820 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1820/)
          HADOOP-10312 changes.text updated in wrong place (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607631)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
            HADOOP-10312 Shell.ExitCodeException to have more useful toString (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607591)
          • /hadoop/common/trunk
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1820 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1820/ ) HADOOP-10312 changes.text updated in wrong place (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607631 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt HADOOP-10312 Shell.ExitCodeException to have more useful toString (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607591 ) /hadoop/common/trunk /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #603 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/603/)
          HADOOP-10312 changes.text updated in wrong place (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607631)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
            HADOOP-10312 Shell.ExitCodeException to have more useful toString (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607591)
          • /hadoop/common/trunk
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #603 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/603/ ) HADOOP-10312 changes.text updated in wrong place (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607631 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt HADOOP-10312 Shell.ExitCodeException to have more useful toString (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607591 ) /hadoop/common/trunk /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1794 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1794/)
          HADOOP-10312 changes.text updated in wrong place (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607631)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1794 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1794/ ) HADOOP-10312 changes.text updated in wrong place (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1607631 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt

            People

            • Assignee:
              Steve Loughran
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development