Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3889

job client tries to use /tasklog interface, but that doesn't exist anymore

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.1, 2.0.0-alpha, 3.0.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None

      Description

      if you specify -Dmapreduce.client.output.filter=SUCCEEDED option when running a job it tries to fetch task logs to print out on the client side from a url like: http://nodemanager:8080/tasklog?plaintext=true&attemptid=attempt_1329857083014_0003_r_000000_0&filter=stdout

      It always errors on this request with: Required param job, map and reduce

      We saw this error when using distcp and the distcp failed. I'm not sure if it is mandatory for distcp or just informational purposes. I'm guessing the latter.

      1. MAPREDUCE-3889.patch
        4 kB
        Devaraj K
      2. MAPREDUCE-3889.patch
        5 kB
        Devaraj K

        Issue Links

          Activity

          Hide
          Vinod Kumar Vavilapalli added a comment -

          Created MAPREDUCE-4362 to see if it is possible to get back the feature of propagating task logs back to JobClient.

          Show
          Vinod Kumar Vavilapalli added a comment - Created MAPREDUCE-4362 to see if it is possible to get back the feature of propagating task logs back to JobClient.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1116 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1116/)
          MAPREDUCE-3889. job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352330)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352330
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1116 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1116/ ) MAPREDUCE-3889 . job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352330) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352330 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #293 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/293/)
          svn merge -c 1352330. FIXES: MAPREDUCE-3889. job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352342)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352342
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #293 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/293/ ) svn merge -c 1352330. FIXES: MAPREDUCE-3889 . job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352342) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352342 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1083 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1083/)
          MAPREDUCE-3889. job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352330)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352330
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1083 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1083/ ) MAPREDUCE-3889 . job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352330) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352330 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2393 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2393/)
          MAPREDUCE-3889. job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352330)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352330
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2393 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2393/ ) MAPREDUCE-3889 . job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352330) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352330 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Hide
          Robert Joseph Evans added a comment -

          Thanks Devaraj,

          Keep up the good work. I put this into branch-0.23, branch-2, and trunk.

          Show
          Robert Joseph Evans added a comment - Thanks Devaraj, Keep up the good work. I put this into branch-0.23, branch-2, and trunk.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2374 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2374/)
          MAPREDUCE-3889. job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352330)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352330
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2374 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2374/ ) MAPREDUCE-3889 . job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352330) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352330 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2444 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2444/)
          MAPREDUCE-3889. job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352330)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352330
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2444 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2444/ ) MAPREDUCE-3889 . job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352330) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352330 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Hide
          Robert Joseph Evans added a comment -

          The patch looks good to me +1. I am happy to check it in, but I would like to see a follow on JIRA to update the client to add back in the functionality.

          Show
          Robert Joseph Evans added a comment - The patch looks good to me +1. I am happy to check it in, but I would like to see a follow on JIRA to update the client to add back in the functionality.
          Hide
          Devaraj K added a comment -
          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.
          

          It is just code removal and there are no tests related to this part of code. I have verified it manually.

          Show
          Devaraj K added a comment - -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. It is just code removal and there are no tests related to this part of code. I have verified it manually.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12531806/MAPREDUCE-3889.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2456//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2456//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531806/MAPREDUCE-3889.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2456//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2456//console This message is automatically generated.
          Hide
          Devaraj K added a comment -

          I have attached patch to disable this. Please review it.

          Show
          Devaraj K added a comment - I have attached patch to disable this. Please review it.
          Hide
          Thomas Graves added a comment -

          I'm fine with disabling it for now.

          Show
          Thomas Graves added a comment - I'm fine with disabling it for now.
          Hide
          Devaraj K added a comment -

          By default this issue is coming from the Job client for all the Jobs which has any failed tasks or if the profiling is enabled(MAPREDUCE-4294).

          When the 400 error prints on the console/job client side, the users may think that due to this error job got failed and starts investigating on this instead of checking the original cause for job failure.

          As Vinod mentioned, I feel it is better to remove/disable this feature as TaskLogServlet support is already removed and there is no info in the Job Client/NM for mapping from task attempt to container id.

          Is there any other way to go ahead for this issue?

          Show
          Devaraj K added a comment - By default this issue is coming from the Job client for all the Jobs which has any failed tasks or if the profiling is enabled( MAPREDUCE-4294 ). When the 400 error prints on the console/job client side, the users may think that due to this error job got failed and starts investigating on this instead of checking the original cause for job failure. As Vinod mentioned, I feel it is better to remove/disable this feature as TaskLogServlet support is already removed and there is no info in the Job Client/NM for mapping from task attempt to container id. Is there any other way to go ahead for this issue?
          Hide
          Thomas Graves added a comment -

          What is the impact of this? Is it crashing the client? Seems like it from the code, in which case we'll need to fix it.

          This is not crashing the client. It just prints the 400 message out on the client if they had a failed task (by default) or task with status by what they set -Dmapreduce.client.output.filter to.

          400 message look like:
          12/02/18 21:32:12 WARN mapreduce.Job: Error reading task output Server returned HTTP response code: 400 for URL:
          http://nodemanager:8080/tasklog?plaintext=true&attemptid=attempt_1329857083014_0003_r_000000_0&filter=stdout

          So as far as I can tell its benign - just possibly confusing to the user and its not actually giving them any of the log information for failed tasks.

          Show
          Thomas Graves added a comment - What is the impact of this? Is it crashing the client? Seems like it from the code, in which case we'll need to fix it. This is not crashing the client. It just prints the 400 message out on the client if they had a failed task (by default) or task with status by what they set -Dmapreduce.client.output.filter to. 400 message look like: 12/02/18 21:32:12 WARN mapreduce.Job: Error reading task output Server returned HTTP response code: 400 for URL: http://nodemanager:8080/tasklog?plaintext=true&attemptid=attempt_1329857083014_0003_r_000000_0&filter=stdout So as far as I can tell its benign - just possibly confusing to the user and its not actually giving them any of the log information for failed tasks.
          Hide
          Daryn Sharp added a comment -

          Yes, this particular bug happens when tasks fail. We probably need another jira for the underlying error that trips this bug:

          2012-02-21 23:13:51,600 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child :
          java.lang.NoSuchMethodError: org.apache.hadoop.tools.DistCp.checkAndClose(Ljava/io/Closeable;)Z
                  at org.apache.hadoop.tools.DistCp$CopyFilesMapper.doCopyFile(DistCp.java:465)
                  at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:519)
                  at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copyWithRetries(DistCp.java:594)
                  at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:650)
                  at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:324)
                  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
                  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399)
                  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
                  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at javax.security.auth.Subject.doAs(Subject.java:396)
                  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
                  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
          

          It's bizarre, because the caller is in the same class as the supposedly missing method.

          Show
          Daryn Sharp added a comment - Yes, this particular bug happens when tasks fail. We probably need another jira for the underlying error that trips this bug: 2012-02-21 23:13:51,600 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.apache.hadoop.tools.DistCp.checkAndClose(Ljava/io/Closeable;)Z at org.apache.hadoop.tools.DistCp$CopyFilesMapper.doCopyFile(DistCp.java:465) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:519) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copyWithRetries(DistCp.java:594) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:650) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:324) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) It's bizarre, because the caller is in the same class as the supposedly missing method.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I just checked the code, the default is only for "FAILED" tasks. So, the error can happen even when there are failed tasks.

          What is the impact of this? Is it crashing the client? Seems like it from the code, in which case we'll need to fix it.

          But the problem is that the client doesn't have a way of mapping TaskAttemptId to the ContainerIDs so that the log-urls can be constructed properly. So may be for the shorter term, we can disable the feature completely.

          Show
          Vinod Kumar Vavilapalli added a comment - I just checked the code, the default is only for "FAILED" tasks. So, the error can happen even when there are failed tasks. What is the impact of this? Is it crashing the client? Seems like it from the code, in which case we'll need to fix it. But the problem is that the client doesn't have a way of mapping TaskAttemptId to the ContainerIDs so that the log-urls can be constructed properly. So may be for the shorter term, we can disable the feature completely.

            People

            • Assignee:
              Devaraj K
              Reporter:
              Thomas Graves
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development