Hive
  1. Hive
  2. HIVE-1579

showJobFailDebugInfo fails job if tasktracker does not respond

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Query Processor
    • Labels:
      None

      Description

      here's the stack trace:

      java.lang.RuntimeException: Error while reading from task log url
      at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
      at org.apache.hadoop.hive.ql.exec.ExecDriver.showJobFailDebugInfo(ExecDriver.java:844)
      at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:624)
      at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:120)
      at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:108)
      at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
      at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:609)
      at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:478)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:356)
      at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:140)
      at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:199)
      at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:316)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
      Caused by: java.io.FileNotFoundException: http://hadoop0062.snc3.facebook.com.:50060/tasklog?taskid=attempt_201008191557_26566\
      _m_000001_3&all=true
      at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1239)
      at java.net.URL.openStream(URL.java:1009)
      at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
      ... 16 more
      Ended Job = job_201008191557_26566 with exception 'java.lang.RuntimeException(Error while reading from task log url)'
      FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask

      this failed a multi hour script.

        Issue Links

          Activity

          Hide
          Richard Williamson added a comment -

          Same issue running Hive 0.6 against CDH3B2... I encountered it after trying to re-run a failed query (GC out of memory orig error which I increased memory on 2nd run). I have tried re-running multiple times with same error and restarting tasktracker on node that keeps giving error (as well as restarting mapred on cluster. The behavior is the query hangs for multiple hours without making progress then fails with this error (same node listed both times in error log).

          Show
          Richard Williamson added a comment - Same issue running Hive 0.6 against CDH3B2... I encountered it after trying to re-run a failed query (GC out of memory orig error which I increased memory on 2nd run). I have tried re-running multiple times with same error and restarting tasktracker on node that keeps giving error (as well as restarting mapred on cluster. The behavior is the query hangs for multiple hours without making progress then fails with this error (same node listed both times in error log).
          Hide
          Richard Williamson added a comment -

          I may have found root cause on this error - when increasing the memory for my failed runs, I left off the dash as follows:
          set mapred.child.java.opts=Xmx1100M;
          When correcting with:
          set mapred.child.java.opts=-Xmx1100M;
          It ran without errors...

          Show
          Richard Williamson added a comment - I may have found root cause on this error - when increasing the memory for my failed runs, I left off the dash as follows: set mapred.child.java.opts=Xmx1100M; When correcting with: set mapred.child.java.opts=-Xmx1100M; It ran without errors...

            People

            • Assignee:
              Paul Yang
              Reporter:
              Joydeep Sen Sarma
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development