Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-3376

Local mode errors point to the wrong file

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Diagnosability, Logging
    • Labels:
      None

      Description

      This is a classic example of a query that fails due to a bad udf in local mode:

      Ended Job = job_local_0001 with errors
      Error during job, obtaining debugging information...
      Execution failed with exit status: 2
      Obtaining error information

      Task failed!
      Task ID:
      Stage-1

      Logs:

      /tmp/gang/hive.log
      FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask

      But actually local mr job in /tmp/gang/gang_20120808181010_b716b5fb-5b8f-49d2-94e9-95bb8f465342.log

      1. hive-3371
        4 kB
        Gang Tim Liu
      2. HIVE-3376.2.patch
        4 kB
        Vaibhav Gumashta

        Activity

        Show
        gangtimliu Gang Tim Liu added a comment - https://reviews.facebook.net/D4575
        Hide
        gangtimliu Gang Tim Liu added a comment -

        While executing in local mode, the process to collect failed job information is a different process from running local MR job. Each process reads a different log4j property file. One points to hive.log and another is MR job log.
        No communication on 2 processes yet on job log. There are some ways like external files etc.
        Consider the bug impact itself (log file location), easy way is to pass MR job stream and pick up log file location and output in CLI stream.

        Show
        gangtimliu Gang Tim Liu added a comment - While executing in local mode, the process to collect failed job information is a different process from running local MR job. Each process reads a different log4j property file. One points to hive.log and another is MR job log. No communication on 2 processes yet on job log. There are some ways like external files etc. Consider the bug impact itself (log file location), easy way is to pass MR job stream and pick up log file location and output in CLI stream.
        Hide
        namit Namit Jain added a comment -

        +1

        Starting tests

        Show
        namit Namit Jain added a comment - +1 Starting tests
        Hide
        cwsteinbach Carl Steinbach added a comment -

        Can you please add a testcase? Thanks.

        Show
        cwsteinbach Carl Steinbach added a comment - Can you please add a testcase? Thanks.
        Hide
        cwsteinbach Carl Steinbach added a comment -

        Please see my other comments on phabricator. Thanks.

        Show
        cwsteinbach Carl Steinbach added a comment - Please see my other comments on phabricator. Thanks.
        Hide
        gangtimliu Gang Tim Liu added a comment -

        @Carl, yes, will add a testcase and address you other comments on phabricator. thanks

        Show
        gangtimliu Gang Tim Liu added a comment - @Carl, yes, will add a testcase and address you other comments on phabricator. thanks
        Hide
        vgumashta Vaibhav Gumashta added a comment -

        Gang Tim Liu Thanks for the work so far - this will be very useful while debugging. Are you still working on this JIRA? If not, I can give it a shot.

        Show
        vgumashta Vaibhav Gumashta added a comment - Gang Tim Liu Thanks for the work so far - this will be very useful while debugging. Are you still working on this JIRA? If not, I can give it a shot.
        Hide
        vgumashta Vaibhav Gumashta added a comment -

        Patch rebased on trunk.

        Show
        vgumashta Vaibhav Gumashta added a comment - Patch rebased on trunk.
        Hide
        hiveqa Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12621866/HIVE-3376.2.patch

        SUCCESS: +1 4903 tests passed

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/823/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/823/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        ATTACHMENT ID: 12621866

        Show
        hiveqa Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12621866/HIVE-3376.2.patch SUCCESS: +1 4903 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/823/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/823/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12621866

          People

          • Assignee:
            gangtimliu Gang Tim Liu
            Reporter:
            gangtimliu Gang Tim Liu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development