Hive
  1. Hive
  2. HIVE-2858

Cache remote map reduce job stack traces for additional logging

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: Logging
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed.

        Activity

        Hide
        Phabricator added a comment -

        kevinwilfong requested code review of "HIVE-2858 [jira] Cache remote map reduce job stack traces for additional logging".
        Reviewers: JIRA

        https://issues.apache.org/jira/browse/HIVE-2858

        Modified the JobDebugger and TaskLogProcessor to parse out stack traces from failed task logs. These are stored in a mapping from job ID to stack traces in the SessionState.

        Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D2223

        AFFECTED FILES
        conf/hive-default.xml.template
        build-common.xml
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out
        ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out
        ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifySessionStateStackTracesHook.java
        ql/src/test/queries/clientnegative/mapreduce_stack_trace.q
        ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff.q
        ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
        ql/src/java/org/apache/hadoop/hive/ql/exec/JobDebugger.java
        ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
        ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java
        ql/src/java/org/apache/hadoop/hive/ql/Driver.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/4875/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - kevinwilfong requested code review of " HIVE-2858 [jira] Cache remote map reduce job stack traces for additional logging". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2858 Modified the JobDebugger and TaskLogProcessor to parse out stack traces from failed task logs. These are stored in a mapping from job ID to stack traces in the SessionState. Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2223 AFFECTED FILES conf/hive-default.xml.template build-common.xml common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifySessionStateStackTracesHook.java ql/src/test/queries/clientnegative/mapreduce_stack_trace.q ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff.q ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ql/src/java/org/apache/hadoop/hive/ql/exec/JobDebugger.java ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java ql/src/java/org/apache/hadoop/hive/ql/Driver.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4875/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-2858 [jira] Cache remote map reduce job stack traces for additional logging".

        can you refresh and re-apply the patch ?
        It is not applying cleanly for me.

        REVISION DETAIL
        https://reviews.facebook.net/D2223

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-2858 [jira] Cache remote map reduce job stack traces for additional logging". can you refresh and re-apply the patch ? It is not applying cleanly for me. REVISION DETAIL https://reviews.facebook.net/D2223
        Hide
        Phabricator added a comment -

        kevinwilfong updated the revision "HIVE-2858 [jira] Cache remote map reduce job stack traces for additional logging".
        Reviewers: JIRA, njain

        Updated and fixed the conflicts.

        REVISION DETAIL
        https://reviews.facebook.net/D2223

        AFFECTED FILES
        conf/hive-default.xml.template
        build-common.xml
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out
        ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out
        ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifySessionStateStackTracesHook.java
        ql/src/test/queries/clientnegative/mapreduce_stack_trace.q
        ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff.q
        ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
        ql/src/java/org/apache/hadoop/hive/ql/exec/JobDebugger.java
        ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
        ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java
        ql/src/java/org/apache/hadoop/hive/ql/Driver.java

        Show
        Phabricator added a comment - kevinwilfong updated the revision " HIVE-2858 [jira] Cache remote map reduce job stack traces for additional logging". Reviewers: JIRA, njain Updated and fixed the conflicts. REVISION DETAIL https://reviews.facebook.net/D2223 AFFECTED FILES conf/hive-default.xml.template build-common.xml common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifySessionStateStackTracesHook.java ql/src/test/queries/clientnegative/mapreduce_stack_trace.q ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff.q ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ql/src/java/org/apache/hadoop/hive/ql/exec/JobDebugger.java ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        Hide
        Namit Jain added a comment -

        Committed. Thanks Kevin

        Show
        Namit Jain added a comment - Committed. Thanks Kevin
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1360 (See https://builds.apache.org/job/Hive-trunk-h0.21/1360/)
        HIVE-2858 Cache remote map reduce job stack traces for additional
        logging (Kevin Wilfong via namit) (Revision 1310583)

        Result = ABORTED
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310583
        Files :

        • /hive/trunk/build-common.xml
        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        • /hive/trunk/conf/hive-default.xml.template
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JobDebugger.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
        • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifySessionStateStackTracesHook.java
        • /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace.q
        • /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff.q
        • /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1360 (See https://builds.apache.org/job/Hive-trunk-h0.21/1360/ ) HIVE-2858 Cache remote map reduce job stack traces for additional logging (Kevin Wilfong via namit) (Revision 1310583) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310583 Files : /hive/trunk/build-common.xml /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JobDebugger.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifySessionStateStackTracesHook.java /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace.q /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff.q /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out
        Hide
        Ashutosh Chauhan added a comment -

        This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

        Show
        Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-2858 Cache remote map reduce job stack traces for additional
        logging (Kevin Wilfong via namit) (Revision 1310583)

        Result = ABORTED
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310583
        Files :

        • /hive/trunk/build-common.xml
        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        • /hive/trunk/conf/hive-default.xml.template
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JobDebugger.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
        • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifySessionStateStackTracesHook.java
        • /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace.q
        • /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff.q
        • /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2858 Cache remote map reduce job stack traces for additional logging (Kevin Wilfong via namit) (Revision 1310583) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310583 Files : /hive/trunk/build-common.xml /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JobDebugger.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifySessionStateStackTracesHook.java /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace.q /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff.q /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out

          People

          • Assignee:
            Kevin Wilfong
            Reporter:
            Kevin Wilfong
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development