Hive
  1. Hive
  2. HIVE-2346

Add hooks to run when execution fails.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, when a query fails, the Post Execution Hooks are not run.

      Adding hooks to be run when a query fails could allow for better logging etc.

      1. HIVE-2346.1.patch.txt
        3 kB
        Kevin Wilfong
      2. HIVE-2346.2.patch.txt
        4 kB
        Kevin Wilfong
      3. HIVE-2346.3.patch.txt
        5 kB
        Kevin Wilfong

        Activity

        Show
        Kevin Wilfong added a comment - https://reviews.apache.org/r/1295/
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1295/
        -----------------------------------------------------------

        Review request for hive and Ning Zhang.

        Summary
        -------

        I added a new type of hook, which will be run when a job fails.

        This addresses bug HIVE-2346.
        https://issues.apache.org/jira/browse/HIVE-2346

        Diffs


        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1153966
        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1153966
        trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 1153966

        Diff: https://reviews.apache.org/r/1295/diff

        Testing
        -------

        I ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed.

        In addition, I created a sample hook, which simply logged that it was run. I verified it was run on a failure, but not when a job succeeded.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1295/ ----------------------------------------------------------- Review request for hive and Ning Zhang. Summary ------- I added a new type of hook, which will be run when a job fails. This addresses bug HIVE-2346 . https://issues.apache.org/jira/browse/HIVE-2346 Diffs trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 1153966 Diff: https://reviews.apache.org/r/1295/diff Testing ------- I ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed. In addition, I created a sample hook, which simply logged that it was run. I verified it was run on a failure, but not when a job succeeded. Thanks, Kevin
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1295/#review1364
        -----------------------------------------------------------

        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        <https://reviews.apache.org/r/1295/#comment3082>

        As a convention, we should declare the variable as interface (List in this case) rather than implementation (ArrayList).

        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        <https://reviews.apache.org/r/1295/#comment3086>

        comments are not correct: post execution --> failure

        • Ning

        On 2011-08-04 19:06:43, Kevin Wilfong wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1295/

        -----------------------------------------------------------

        (Updated 2011-08-04 19:06:43)

        Review request for hive and Ning Zhang.

        Summary

        -------

        I added a new type of hook, which will be run when a job fails.

        This addresses bug HIVE-2346.

        https://issues.apache.org/jira/browse/HIVE-2346

        Diffs

        -----

        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1153966

        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1153966

        trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 1153966

        Diff: https://reviews.apache.org/r/1295/diff

        Testing

        -------

        I ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed.

        In addition, I created a sample hook, which simply logged that it was run. I verified it was run on a failure, but not when a job succeeded.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1295/#review1364 ----------------------------------------------------------- trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java < https://reviews.apache.org/r/1295/#comment3082 > As a convention, we should declare the variable as interface (List in this case) rather than implementation (ArrayList). trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java < https://reviews.apache.org/r/1295/#comment3086 > comments are not correct: post execution --> failure Ning On 2011-08-04 19:06:43, Kevin Wilfong wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1295/ ----------------------------------------------------------- (Updated 2011-08-04 19:06:43) Review request for hive and Ning Zhang. Summary ------- I added a new type of hook, which will be run when a job fails. This addresses bug HIVE-2346 . https://issues.apache.org/jira/browse/HIVE-2346 Diffs ----- trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 1153966 Diff: https://reviews.apache.org/r/1295/diff Testing ------- I ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed. In addition, I created a sample hook, which simply logged that it was run. I verified it was run on a failure, but not when a job succeeded. Thanks, Kevin
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1295/
        -----------------------------------------------------------

        (Updated 2011-08-09 22:01:28.619424)

        Review request for hive and Ning Zhang.

        Changes
        -------

        Made the changes requested by nzhang.

        Regarding the change ArrayList --> List, I had copied that code from the analogous methods for Post and Pre Exec hooks. So I made the corresponding changes in those methods too.

        Summary
        -------

        I added a new type of hook, which will be run when a job fails.

        This addresses bug HIVE-2346.
        https://issues.apache.org/jira/browse/HIVE-2346

        Diffs (updated)


        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1155569
        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1155569
        trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 1155569

        Diff: https://reviews.apache.org/r/1295/diff

        Testing
        -------

        I ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed.

        In addition, I created a sample hook, which simply logged that it was run. I verified it was run on a failure, but not when a job succeeded.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1295/ ----------------------------------------------------------- (Updated 2011-08-09 22:01:28.619424) Review request for hive and Ning Zhang. Changes ------- Made the changes requested by nzhang. Regarding the change ArrayList --> List, I had copied that code from the analogous methods for Post and Pre Exec hooks. So I made the corresponding changes in those methods too. Summary ------- I added a new type of hook, which will be run when a job fails. This addresses bug HIVE-2346 . https://issues.apache.org/jira/browse/HIVE-2346 Diffs (updated) trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1155569 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1155569 trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 1155569 Diff: https://reviews.apache.org/r/1295/diff Testing ------- I ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed. In addition, I created a sample hook, which simply logged that it was run. I verified it was run on a failure, but not when a job succeeded. Thanks, Kevin
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1295/#review1370
        -----------------------------------------------------------

        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        <https://reviews.apache.org/r/1295/#comment3144>

        Kevin, can you add this parameter to conf/hive-default.xml? We should add all newly introduced parameters to that file.

        Otherwise it looks good to me.

        • Ning

        On 2011-08-09 22:01:28, Kevin Wilfong wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/1295/

        -----------------------------------------------------------

        (Updated 2011-08-09 22:01:28)

        Review request for hive and Ning Zhang.

        Summary

        -------

        I added a new type of hook, which will be run when a job fails.

        This addresses bug HIVE-2346.

        https://issues.apache.org/jira/browse/HIVE-2346

        Diffs

        -----

        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1155569

        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1155569

        trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 1155569

        Diff: https://reviews.apache.org/r/1295/diff

        Testing

        -------

        I ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed.

        In addition, I created a sample hook, which simply logged that it was run. I verified it was run on a failure, but not when a job succeeded.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1295/#review1370 ----------------------------------------------------------- trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java < https://reviews.apache.org/r/1295/#comment3144 > Kevin, can you add this parameter to conf/hive-default.xml? We should add all newly introduced parameters to that file. Otherwise it looks good to me. Ning On 2011-08-09 22:01:28, Kevin Wilfong wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1295/ ----------------------------------------------------------- (Updated 2011-08-09 22:01:28) Review request for hive and Ning Zhang. Summary ------- I added a new type of hook, which will be run when a job fails. This addresses bug HIVE-2346 . https://issues.apache.org/jira/browse/HIVE-2346 Diffs ----- trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1155569 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1155569 trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 1155569 Diff: https://reviews.apache.org/r/1295/diff Testing ------- I ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed. In addition, I created a sample hook, which simply logged that it was run. I verified it was run on a failure, but not when a job succeeded. Thanks, Kevin
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1295/
        -----------------------------------------------------------

        (Updated 2011-08-09 23:44:00.972625)

        Review request for hive and Ning Zhang.

        Changes
        -------

        Added the hive.exec.failure.hooks property to hive-default.xml

        Summary
        -------

        I added a new type of hook, which will be run when a job fails.

        This addresses bug HIVE-2346.
        https://issues.apache.org/jira/browse/HIVE-2346

        Diffs (updated)


        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1155569
        trunk/conf/hive-default.xml 1155569
        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1155569
        trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 1155569

        Diff: https://reviews.apache.org/r/1295/diff

        Testing
        -------

        I ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed.

        In addition, I created a sample hook, which simply logged that it was run. I verified it was run on a failure, but not when a job succeeded.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1295/ ----------------------------------------------------------- (Updated 2011-08-09 23:44:00.972625) Review request for hive and Ning Zhang. Changes ------- Added the hive.exec.failure.hooks property to hive-default.xml Summary ------- I added a new type of hook, which will be run when a job fails. This addresses bug HIVE-2346 . https://issues.apache.org/jira/browse/HIVE-2346 Diffs (updated) trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1155569 trunk/conf/hive-default.xml 1155569 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1155569 trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 1155569 Diff: https://reviews.apache.org/r/1295/diff Testing ------- I ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed. In addition, I created a sample hook, which simply logged that it was run. I verified it was run on a failure, but not when a job succeeded. Thanks, Kevin
        Hide
        Ning Zhang added a comment -

        +1. Will commit if tests pass.

        Show
        Ning Zhang added a comment - +1. Will commit if tests pass.
        Hide
        Ning Zhang added a comment -

        Committed. Thanks Kevin!

        Show
        Ning Zhang added a comment - Committed. Thanks Kevin!
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #887 (See https://builds.apache.org/job/Hive-trunk-h0.21/887/)
        HIVE-2346. Add hooks to run when execution fails. (Kevin Wilfong via Ning Zhang)

        nzhang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1156480
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        • /hive/trunk/conf/hive-default.xml
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #887 (See https://builds.apache.org/job/Hive-trunk-h0.21/887/ ) HIVE-2346 . Add hooks to run when execution fails. (Kevin Wilfong via Ning Zhang) nzhang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1156480 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/conf/hive-default.xml

          People

          • Assignee:
            Kevin Wilfong
            Reporter:
            Kevin Wilfong
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development