Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 3.0.0, 2.0.2-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: client, task
    • Labels:
      None

      Description

      It's exceedingly difficult to debug token issues with a job. Sometimes tokens appear to be missing, or have the wrong service, etc. It would be much easier if job submission logged the tokens submitted, and if tasks logged the tokens they received.

      1. MAPREDUCE-4614.patch
        2 kB
        Daryn Sharp
      2. MAPREDUCE-4614.branch-23.patch
        2 kB
        Daryn Sharp

        Issue Links

          Activity

          Hide
          Daryn Sharp added a comment -

          This patch helped find the problem in HDFS-3873.

          Show
          Daryn Sharp added a comment - This patch helped find the problem in HDFS-3873 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12543190/MAPREDUCE-4614.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2797//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2797//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543190/MAPREDUCE-4614.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2797//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2797//console This message is automatically generated.
          Hide
          Siddharth Seth added a comment -

          Looks good. Would prefer the client side messages to be at a DEBUG level. If this is really required for debugging (issues which are tough to reproduce etc), maybe we can have them at INFO level while stabilizing the system, and change them to DEBUG at a later point. Otherwise, DEBUG level + a change in the client log level should work.

          Show
          Siddharth Seth added a comment - Looks good. Would prefer the client side messages to be at a DEBUG level. If this is really required for debugging (issues which are tough to reproduce etc), maybe we can have them at INFO level while stabilizing the system, and change them to DEBUG at a later point. Otherwise, DEBUG level + a change in the client log level should work.
          Hide
          Daryn Sharp added a comment -

          I'm ok with INFO during stabilization. As you mention, the problem with client DEBUG log level is we miss the bugs "in the act". Plus we've had enough job submission issues lately that affect oozie which are painfully hard to debug. It's been indispensable to see the submitted tokens, see if the launcher task acquired them, whether the launcher task's job submission sent them along, followed by whether they popped out in the other job.

          Show
          Daryn Sharp added a comment - I'm ok with INFO during stabilization. As you mention, the problem with client DEBUG log level is we miss the bugs "in the act". Plus we've had enough job submission issues lately that affect oozie which are painfully hard to debug. It's been indispensable to see the submitted tokens, see if the launcher task acquired them, whether the launcher task's job submission sent them along, followed by whether they popped out in the other job.
          Hide
          Robert Joseph Evans added a comment -

          The changes look OK to me +1. Please file a JIRA to move the log level back to debug on the job submission ones so we don't forget about it. Although it is chatty enough I don't think we would forget about it.

          +1. I'll check it in.

          Show
          Robert Joseph Evans added a comment - The changes look OK to me +1. Please file a JIRA to move the log level back to debug on the job submission ones so we don't forget about it. Although it is chatty enough I don't think we would forget about it. +1. I'll check it in.
          Hide
          Robert Joseph Evans added a comment -

          Thanks Daryn,

          I put this into trunk, branch-2, and branch-0.23

          Show
          Robert Joseph Evans added a comment - Thanks Daryn, I put this into trunk, branch-2, and branch-0.23
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2728 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2728/)
          MAPREDUCE-4614. Simplify debugging a job's tokens (daryn via bobby) (Revision 1379595)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379595
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2728 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2728/ ) MAPREDUCE-4614 . Simplify debugging a job's tokens (daryn via bobby) (Revision 1379595) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379595 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2665 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2665/)
          MAPREDUCE-4614. Simplify debugging a job's tokens (daryn via bobby) (Revision 1379595)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379595
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2665 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2665/ ) MAPREDUCE-4614 . Simplify debugging a job's tokens (daryn via bobby) (Revision 1379595) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379595 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2691 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2691/)
          MAPREDUCE-4614. Simplify debugging a job's tokens (daryn via bobby) (Revision 1379595)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379595
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2691 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2691/ ) MAPREDUCE-4614 . Simplify debugging a job's tokens (daryn via bobby) (Revision 1379595) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379595 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #361 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/361/)
          MAPREDUCE-4614. Simplify debugging a job's tokens (daryn via bobby) (Revision 1379597)

          Result = UNSTABLE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379597
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #361 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/361/ ) MAPREDUCE-4614 . Simplify debugging a job's tokens (daryn via bobby) (Revision 1379597) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379597 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1152 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1152/)
          MAPREDUCE-4614. Simplify debugging a job's tokens (daryn via bobby) (Revision 1379595)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379595
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1152 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1152/ ) MAPREDUCE-4614 . Simplify debugging a job's tokens (daryn via bobby) (Revision 1379595) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379595 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1183 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1183/)
          MAPREDUCE-4614. Simplify debugging a job's tokens (daryn via bobby) (Revision 1379595)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379595
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1183 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1183/ ) MAPREDUCE-4614 . Simplify debugging a job's tokens (daryn via bobby) (Revision 1379595) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379595 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development