Hive
  1. Hive
  2. HIVE-2619

Add hook to run in metastore's endFunction which can collect more fb303 counters

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Create the potential for hooks to run in the endFunction method of HMSHandler which take the name of a function and whether or not it succeeded. Also, override getCounters from fb303 to allow these hooks to add counters which they collect, should this be desired. These hooks can be similar to EventListeners, but they should be more generic.

        Activity

        Kevin Wilfong created issue -
        Kevin Wilfong made changes -
        Field Original Value New Value
        Attachment HIVE-2619.1.patch.txt [ 12505701 ]
        Hide
        Namit Jain added a comment -
        Show
        Namit Jain added a comment - https://reviews.facebook.net/D561
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-2619 [jira] Add hook to run in meatastore's endFunction which can collect more fb303 counters".

        REVISION DETAIL
        https://reviews.facebook.net/D561

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-2619 [jira] Add hook to run in meatastore's endFunction which can collect more fb303 counters". REVISION DETAIL https://reviews.facebook.net/D561
        Hide
        Kevin Wilfong added a comment -

        I also created a diff here https://reviews.facebook.net/D555

        I thought it was supposed to update here automatically.

        Show
        Kevin Wilfong added a comment - I also created a diff here https://reviews.facebook.net/D555 I thought it was supposed to update here automatically.
        Hide
        Namit Jain added a comment -

        I thought so too - but it did not for some reason.
        John, do you know ?

        I did:

        ant arc-setup
        arc diff --jira HIVE-2619

        Show
        Namit Jain added a comment - I thought so too - but it did not for some reason. John, do you know ? I did: ant arc-setup arc diff --jira HIVE-2619
        Hide
        Namit Jain added a comment -

        It did show up for me - a little delayed. I should have been more patient.

        Show
        Namit Jain added a comment - It did show up for me - a little delayed. I should have been more patient.
        Hide
        John Sichi added a comment -

        It should have added JIRA as one of the reviewers on the diff (that's how the comments get forwarded). Did you remove JIRA from the reviewers list?

        You can add it back on manually by editing the diff, and then arc diff again to update the diff and it should add info here. +Marek.

        Mmmmmm...meatastore. Can I get bacon with that?

        Show
        John Sichi added a comment - It should have added JIRA as one of the reviewers on the diff (that's how the comments get forwarded). Did you remove JIRA from the reviewers list? You can add it back on manually by editing the diff, and then arc diff again to update the diff and it should add info here. +Marek. Mmmmmm...meatastore. Can I get bacon with that?
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-2619 [jira] Add hook to run in meatastore's endFunction which can collect more fb303 counters".

        INLINE COMMENTS
        metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionListener.java:41 Looks good - do you want to pass a metastoreContext to the endFunction.

        This can be a empty class right now, but in future if we need to add
        anything, it would be useful

        REVISION DETAIL
        https://reviews.facebook.net/D561

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-2619 [jira] Add hook to run in meatastore's endFunction which can collect more fb303 counters". INLINE COMMENTS metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionListener.java:41 Looks good - do you want to pass a metastoreContext to the endFunction. This can be a empty class right now, but in future if we need to add anything, it would be useful REVISION DETAIL https://reviews.facebook.net/D561
        Hide
        Phabricator added a comment -

        njain requested code review of "HIVE-2619 [jira] Add hook to run in meatastore's endFunction which can collect more fb303 counters".
        Reviewers: JIRA

        HIVE-2619

        Create the potential for hooks to run in the endFunction method of HMSHandler which take the name of a function and whether or not it succeeded. Also, override getCounters from fb303 to allow these hooks to add counters which they collect, should this be desired. These hooks can be similar to EventListeners, but they should be more generic.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D561

        AFFECTED FILES
        conf/hive-default.xml
        metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionListener.java
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/1251/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - njain requested code review of " HIVE-2619 [jira] Add hook to run in meatastore's endFunction which can collect more fb303 counters". Reviewers: JIRA HIVE-2619 Create the potential for hooks to run in the endFunction method of HMSHandler which take the name of a function and whether or not it succeeded. Also, override getCounters from fb303 to allow these hooks to add counters which they collect, should this be desired. These hooks can be similar to EventListeners, but they should be more generic. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D561 AFFECTED FILES conf/hive-default.xml metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionListener.java metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java common/src/java/org/apache/hadoop/hive/conf/HiveConf.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/1251/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Phabricator made changes -
        Attachment HIVE-2619.D561.1.patch [ 12505725 ]
        Hide
        Phabricator added a comment -

        kevinwilfong requested code review of "HIVE-2619 [jira] Add hook to run in meatastore's endFunction which can collect more fb303 counters".
        Reviewers: njain, heyongqiang, JIRA

        I added a new type of listener called MetaStoreEndFunctionListener, which is called by endFunction in HMSHandler and takes the method's name, and whether it succeeded. I also overrode FacebookBase's getCounters method to include counters from these listeners.

        These new listeners are configurable in the same way as EventListeners.

        I had to modify endFunction and all calls to it to pass in whether or not the method succeeded.

        TEST PLAN
        I created an implementation of MetaStoreEndFunctionListener and verified it ran without exceptions, and it could add fb303 counters.

        I also ran the unit tests.

        REVISION DETAIL
        https://reviews.facebook.net/D555

        AFFECTED FILES
        conf/hive-default.xml
        metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionListener.java
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java

        Show
        Phabricator added a comment - kevinwilfong requested code review of " HIVE-2619 [jira] Add hook to run in meatastore's endFunction which can collect more fb303 counters". Reviewers: njain, heyongqiang, JIRA I added a new type of listener called MetaStoreEndFunctionListener, which is called by endFunction in HMSHandler and takes the method's name, and whether it succeeded. I also overrode FacebookBase's getCounters method to include counters from these listeners. These new listeners are configurable in the same way as EventListeners. I had to modify endFunction and all calls to it to pass in whether or not the method succeeded. TEST PLAN I created an implementation of MetaStoreEndFunctionListener and verified it ran without exceptions, and it could add fb303 counters. I also ran the unit tests. REVISION DETAIL https://reviews.facebook.net/D555 AFFECTED FILES conf/hive-default.xml metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionListener.java metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        Phabricator made changes -
        Attachment HIVE-2619.D555.1.patch [ 12505790 ]
        Hide
        Phabricator added a comment -

        kevinwilfong updated the revision "HIVE-2619 [jira] Add hook to run in meatastore's endFunction which can collect more fb303 counters".
        Reviewers: JIRA

        Namit made the following comment here https://reviews.facebook.net/D561

        "Looks good - do you want to pass a metastoreContext to the endFunction.

        This can be a empty class right now, but in future if we need to add
        anything, it would be useful"

        That's a good idea, we can put perf data there later.

        REVISION DETAIL
        https://reviews.facebook.net/D555

        AFFECTED FILES
        conf/hive-default.xml
        metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionContext.java
        metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionListener.java
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java

        Show
        Phabricator added a comment - kevinwilfong updated the revision " HIVE-2619 [jira] Add hook to run in meatastore's endFunction which can collect more fb303 counters". Reviewers: JIRA Namit made the following comment here https://reviews.facebook.net/D561 "Looks good - do you want to pass a metastoreContext to the endFunction. This can be a empty class right now, but in future if we need to add anything, it would be useful" That's a good idea, we can put perf data there later. REVISION DETAIL https://reviews.facebook.net/D555 AFFECTED FILES conf/hive-default.xml metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionContext.java metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionListener.java metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        Phabricator made changes -
        Attachment HIVE-2619.D555.2.patch [ 12505791 ]
        Hide
        Namit Jain added a comment -

        +1

        Show
        Namit Jain added a comment - +1
        Namit Jain made changes -
        Summary Add hook to run in meatastore's endFunction which can collect more fb303 counters Add hook to run in metastore's endFunction which can collect more fb303 counters
        Hide
        Namit Jain added a comment -

        Committed. Thanks Kevin

        Show
        Namit Jain added a comment - Committed. Thanks Kevin
        Namit Jain made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1118 (See https://builds.apache.org/job/Hive-trunk-h0.21/1118/)
        HIVE-2619 Add hook to run in metastore's endFunction which can collect more fb303 counters
        (Kevin Wilfong via namit)

        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209368
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        • /hive/trunk/conf/hive-default.xml
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionContext.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionListener.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1118 (See https://builds.apache.org/job/Hive-trunk-h0.21/1118/ ) HIVE-2619 Add hook to run in metastore's endFunction which can collect more fb303 counters (Kevin Wilfong via namit) namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209368 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionContext.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEndFunctionListener.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
        Carl Steinbach made changes -
        Fix Version/s 0.9.0 [ 12317742 ]
        Carl Steinbach made changes -
        Fix Version/s 0.8.0 [ 12316178 ]
        Carl Steinbach made changes -
        Fix Version/s 0.9.0 [ 12317742 ]
        Carl Steinbach made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Kevin Wilfong
            Reporter:
            Kevin Wilfong
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development