Hive
  1. Hive
  2. HIVE-2871

Add a new hook to run at the beginning and end of the Driver.run method

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Driver.run is the highest level method which all queries go through, whether they come from Hive Server, the CLI, or any other entry. We also do not have any hooks before the compilation method is called, and having hooks in Driver.run would provide this. Having hooks in Driver.run will allow, for example, being able to overwrite config values used throughout query processing, including compilation, and at the other end, cleaning up any resources/logging any final values just before returning to the user.

        Activity

        Hide
        Phabricator added a comment -

        kevinwilfong requested code review of "HIVE-2871 [jira] Add a new hook to run at the beginning and end of the Driver.run method".
        Reviewers: JIRA

        https://issues.apache.org/jira/browse/HIVE-2871

        Added a new hook which runs at the beginning and end of the Driver.run method. It has pre and post execution methods. I do not provide any implementations as part of this diff.

        Driver.run is the highest level method which all queries go through, whether they come from Hive Server, the CLI, or any other entry. We also do not have any hooks before the compilation method is called, and having hooks in Driver.run would provide this. Having hooks in Driver.run will allow, for example, being able to overwrite config values used throughout query processing, including compilation, and at the other end, cleaning up any resources/logging any final values just before returning to the user.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D2331

        AFFECTED FILES
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientpositive/hook_order.q.out
        ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyHooksRunInOrder.java
        ql/src/test/queries/clientpositive/hook_order.q
        ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHook.java
        ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContextImpl.java
        ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContext.java
        ql/src/java/org/apache/hadoop/hive/ql/Driver.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/5163/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - kevinwilfong requested code review of " HIVE-2871 [jira] Add a new hook to run at the beginning and end of the Driver.run method". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2871 Added a new hook which runs at the beginning and end of the Driver.run method. It has pre and post execution methods. I do not provide any implementations as part of this diff. Driver.run is the highest level method which all queries go through, whether they come from Hive Server, the CLI, or any other entry. We also do not have any hooks before the compilation method is called, and having hooks in Driver.run would provide this. Having hooks in Driver.run will allow, for example, being able to overwrite config values used throughout query processing, including compilation, and at the other end, cleaning up any resources/logging any final values just before returning to the user. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2331 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientpositive/hook_order.q.out ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyHooksRunInOrder.java ql/src/test/queries/clientpositive/hook_order.q ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHook.java ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContextImpl.java ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContext.java ql/src/java/org/apache/hadoop/hive/ql/Driver.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/5163/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-2871 [jira] Add a new hook to run at the beginning and end of the Driver.run method".

        INLINE COMMENTS
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:578 We should explicitly state here that the hooks will be run in order.

        Also, add a comment in hive-default.xml.default explaining this
        ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContextImpl.java:25 Ado you want to add the sessionState also ?

        REVISION DETAIL
        https://reviews.facebook.net/D2331

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-2871 [jira] Add a new hook to run at the beginning and end of the Driver.run method". INLINE COMMENTS common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:578 We should explicitly state here that the hooks will be run in order. Also, add a comment in hive-default.xml.default explaining this ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContextImpl.java:25 Ado you want to add the sessionState also ? REVISION DETAIL https://reviews.facebook.net/D2331
        Hide
        Phabricator added a comment -

        kevinwilfong updated the revision "HIVE-2871 [jira] Add a new hook to run at the beginning and end of the Driver.run method".
        Reviewers: JIRA, njain

        Expanded comment in HiveConf, and added entry in hive-default.xml.template

        REVISION DETAIL
        https://reviews.facebook.net/D2331

        AFFECTED FILES
        conf/hive-default.xml.template
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientpositive/hook_order.q.out
        ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyHooksRunInOrder.java
        ql/src/test/queries/clientpositive/hook_order.q
        ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHook.java
        ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContextImpl.java
        ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContext.java
        ql/src/java/org/apache/hadoop/hive/ql/Driver.java

        Show
        Phabricator added a comment - kevinwilfong updated the revision " HIVE-2871 [jira] Add a new hook to run at the beginning and end of the Driver.run method". Reviewers: JIRA, njain Expanded comment in HiveConf, and added entry in hive-default.xml.template REVISION DETAIL https://reviews.facebook.net/D2331 AFFECTED FILES conf/hive-default.xml.template common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientpositive/hook_order.q.out ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyHooksRunInOrder.java ql/src/test/queries/clientpositive/hook_order.q ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHook.java ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContextImpl.java ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContext.java ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        Hide
        Phabricator added a comment -

        kevinwilfong has commented on the revision "HIVE-2871 [jira] Add a new hook to run at the beginning and end of the Driver.run method".

        INLINE COMMENTS
        ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContextImpl.java:25 The reason I didn't add it is that in other hook contexts (HookContext and HiveSemanticAnalyzerHookContext) the SessionState is not included. Hooks can get it via SessionState.get()

        I don't really see the advantage of adding it, but if you feel strongly about this I can.

        REVISION DETAIL
        https://reviews.facebook.net/D2331

        Show
        Phabricator added a comment - kevinwilfong has commented on the revision " HIVE-2871 [jira] Add a new hook to run at the beginning and end of the Driver.run method". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContextImpl.java:25 The reason I didn't add it is that in other hook contexts (HookContext and HiveSemanticAnalyzerHookContext) the SessionState is not included. Hooks can get it via SessionState.get() I don't really see the advantage of adding it, but if you feel strongly about this I can. REVISION DETAIL https://reviews.facebook.net/D2331
        Hide
        Namit Jain added a comment -

        +1

        running tests

        Show
        Namit Jain added a comment - +1 running tests
        Hide
        Phabricator added a comment -

        njain has accepted the revision "HIVE-2871 [jira] Add a new hook to run at the beginning and end of the Driver.run method".

        REVISION DETAIL
        https://reviews.facebook.net/D2331

        BRANCH
        svn

        Show
        Phabricator added a comment - njain has accepted the revision " HIVE-2871 [jira] Add a new hook to run at the beginning and end of the Driver.run method". REVISION DETAIL https://reviews.facebook.net/D2331 BRANCH svn
        Hide
        Phabricator added a comment -

        njain has requested changes to the revision "HIVE-2871 [jira] Add a new hook to run at the beginning and end of the Driver.run method".

        INLINE COMMENTS
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:34 Compile is failing, since these imports have been removed

        REVISION DETAIL
        https://reviews.facebook.net/D2331

        BRANCH
        svn

        Show
        Phabricator added a comment - njain has requested changes to the revision " HIVE-2871 [jira] Add a new hook to run at the beginning and end of the Driver.run method". INLINE COMMENTS common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:34 Compile is failing, since these imports have been removed REVISION DETAIL https://reviews.facebook.net/D2331 BRANCH svn
        Hide
        Phabricator added a comment -

        kevinwilfong updated the revision "HIVE-2871 [jira] Add a new hook to run at the beginning and end of the Driver.run method".
        Reviewers: JIRA, njain

        So sorry about that, my eclipse tends to remove imports if I haven't setup the build path properly. Ran ant clean package, so it should be good.

        REVISION DETAIL
        https://reviews.facebook.net/D2331

        AFFECTED FILES
        conf/hive-default.xml.template
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientpositive/hook_order.q.out
        ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyHooksRunInOrder.java
        ql/src/test/queries/clientpositive/hook_order.q
        ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHook.java
        ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContextImpl.java
        ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContext.java
        ql/src/java/org/apache/hadoop/hive/ql/Driver.java

        Show
        Phabricator added a comment - kevinwilfong updated the revision " HIVE-2871 [jira] Add a new hook to run at the beginning and end of the Driver.run method". Reviewers: JIRA, njain So sorry about that, my eclipse tends to remove imports if I haven't setup the build path properly. Ran ant clean package, so it should be good. REVISION DETAIL https://reviews.facebook.net/D2331 AFFECTED FILES conf/hive-default.xml.template common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientpositive/hook_order.q.out ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyHooksRunInOrder.java ql/src/test/queries/clientpositive/hook_order.q ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHook.java ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContextImpl.java ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContext.java ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        Hide
        Namit Jain added a comment -

        Committed. Thanks Kevin

        Show
        Namit Jain added a comment - Committed. Thanks Kevin
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1316 (See https://builds.apache.org/job/Hive-trunk-h0.21/1316/)
        HIVE-2871 Add a new hook to run at the beginning and end of the Driver.run method
        (Kevin Wilfong via namit) (Revision 1301610)

        Result = FAILURE
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301610
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        • /hive/trunk/conf/hive-default.xml.template
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHook.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContext.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContextImpl.java
        • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyHooksRunInOrder.java
        • /hive/trunk/ql/src/test/queries/clientpositive/hook_order.q
        • /hive/trunk/ql/src/test/results/clientpositive/hook_order.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1316 (See https://builds.apache.org/job/Hive-trunk-h0.21/1316/ ) HIVE-2871 Add a new hook to run at the beginning and end of the Driver.run method (Kevin Wilfong via namit) (Revision 1301610) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301610 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHook.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContext.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContextImpl.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyHooksRunInOrder.java /hive/trunk/ql/src/test/queries/clientpositive/hook_order.q /hive/trunk/ql/src/test/results/clientpositive/hook_order.q.out
        Hide
        Ashutosh Chauhan added a comment -

        This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

        Show
        Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-2871 Add a new hook to run at the beginning and end of the Driver.run method
        (Kevin Wilfong via namit) (Revision 1301610)

        Result = ABORTED
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301610
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        • /hive/trunk/conf/hive-default.xml.template
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHook.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContext.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContextImpl.java
        • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyHooksRunInOrder.java
        • /hive/trunk/ql/src/test/queries/clientpositive/hook_order.q
        • /hive/trunk/ql/src/test/results/clientpositive/hook_order.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2871 Add a new hook to run at the beginning and end of the Driver.run method (Kevin Wilfong via namit) (Revision 1301610) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301610 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHook.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContext.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/HiveDriverRunHookContextImpl.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyHooksRunInOrder.java /hive/trunk/ql/src/test/queries/clientpositive/hook_order.q /hive/trunk/ql/src/test/results/clientpositive/hook_order.q.out

          People

          • Assignee:
            Kevin Wilfong
            Reporter:
            Kevin Wilfong
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development