Hive
  1. Hive
  2. HIVE-3400

Add Retries to Hive MetaStore Connections

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Metastore
    • Labels:

      Description

      Currently, when using Thrift to access the MetaStore, if the Thrift host dies, there is no mechanism to reconnect to some other host even if the MetaStore URIs variable in the Conf contains multiple hosts. Hive should retry and reconnect rather than throwing a communication link error.

      1. HIVE-3400.3.patch.txt
        38 kB
        Bhushan Mandhani
      2. HIVE-3400.2.patch.txt
        38 kB
        Bhushan Mandhani
      3. HIVE-3400.1.patch.txt
        11 kB
        Bhushan Mandhani

        Activity

        Ashutosh Chauhan made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Ashutosh Chauhan added a comment -

        This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-3400 : Add Retries to Hive MetaStore Connections (Bhushan Mandhani via Ashutosh Chauhan) (Revision 1418190)

        Result = ABORTED
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1418190
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        • /hive/trunk/conf/hive-default.xml.template
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingRawStore.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreAuthorization.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEndFunctionListener.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3400 : Add Retries to Hive MetaStore Connections (Bhushan Mandhani via Ashutosh Chauhan) (Revision 1418190) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1418190 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingRawStore.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreAuthorization.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEndFunctionListener.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1840 (See https://builds.apache.org/job/Hive-trunk-h0.21/1840/)
        HIVE-3400 : Add Retries to Hive MetaStore Connections (Bhushan Mandhani via Ashutosh Chauhan) (Revision 1418190)

        Result = FAILURE
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1418190
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        • /hive/trunk/conf/hive-default.xml.template
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingRawStore.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreAuthorization.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEndFunctionListener.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1840 (See https://builds.apache.org/job/Hive-trunk-h0.21/1840/ ) HIVE-3400 : Add Retries to Hive MetaStore Connections (Bhushan Mandhani via Ashutosh Chauhan) (Revision 1418190) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1418190 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingRawStore.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreAuthorization.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEndFunctionListener.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
        Ashutosh Chauhan made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.10.0 [ 12320745 ]
        Resolution Fixed [ 1 ]
        Hide
        Ashutosh Chauhan added a comment -

        Committed to trunk and 0.10. Thanks, Bhushan!

        Show
        Ashutosh Chauhan added a comment - Committed to trunk and 0.10. Thanks, Bhushan!
        Bhushan Mandhani made changes -
        Attachment HIVE-3400.3.patch.txt [ 12556212 ]
        Hide
        Ashutosh Chauhan added a comment -

        Bhushan,
        Patch wasn't applying cleanly on trunk, so I had to resolve some conflicts. I did that and ran tests and all the test cases passed. Great progress!
        But, I have few minor comments which I left on phabricator. If you can take care of those and generate a patch against trunk and attach here, I will take care of committing and closing this issue.

        Show
        Ashutosh Chauhan added a comment - Bhushan, Patch wasn't applying cleanly on trunk, so I had to resolve some conflicts. I did that and ran tests and all the test cases passed. Great progress! But, I have few minor comments which I left on phabricator. If you can take care of those and generate a patch against trunk and attach here, I will take care of committing and closing this issue.
        Hide
        Ashutosh Chauhan added a comment -

        +1 Running tests now.

        Show
        Ashutosh Chauhan added a comment - +1 Running tests now.
        Hide
        Bhushan Mandhani added a comment -

        Ashutosh, we no longer need HIVE-3612. Jean is about to abandon that diff. I think we should keep these RetryingRawStore changes here since RetryingHMSHandler already catches JDOExceptions. But I can take it out if you prefer that.

        Show
        Bhushan Mandhani added a comment - Ashutosh, we no longer need HIVE-3612 . Jean is about to abandon that diff. I think we should keep these RetryingRawStore changes here since RetryingHMSHandler already catches JDOExceptions. But I can take it out if you prefer that.
        Hide
        Ashutosh Chauhan added a comment -

        Bhushan, You have some changes in RetryingRawStore.java which looks like are unrelated to this patch. It seems like HIVE-3612 is tracking similar enhancements. Will you mind taking those changes out and put on in HIVE-3612?

        Show
        Ashutosh Chauhan added a comment - Bhushan, You have some changes in RetryingRawStore.java which looks like are unrelated to this patch. It seems like HIVE-3612 is tracking similar enhancements. Will you mind taking those changes out and put on in HIVE-3612 ?
        Hide
        Bhushan Mandhani added a comment -

        Ashutosh, I've uploaded and submitted the latest patch. Thanks.

        Show
        Bhushan Mandhani added a comment - Ashutosh, I've uploaded and submitted the latest patch. Thanks.
        Bhushan Mandhani made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Labels metastore
        Bhushan Mandhani made changes -
        Attachment HIVE-3400.2.patch.txt [ 12555555 ]
        Hide
        Bhushan Mandhani added a comment -

        Yes, I'll do "Submit Patch" after making one key change that Carl pointed out. Working on that now.

        Show
        Bhushan Mandhani added a comment - Yes, I'll do "Submit Patch" after making one key change that Carl pointed out. Working on that now.
        Hide
        Ashutosh Chauhan added a comment -

        Bhushan, Can you upload the latest patch on the jira too ?

        Show
        Ashutosh Chauhan added a comment - Bhushan, Can you upload the latest patch on the jira too ?
        Hide
        Ashutosh Chauhan added a comment -

        Bhushan, are you still working on this one ?

        Show
        Ashutosh Chauhan added a comment - Bhushan, are you still working on this one ?
        Carl Steinbach made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Carl Steinbach added a comment -

        More comments on phabricator.

        Show
        Carl Steinbach added a comment - More comments on phabricator.
        Bhushan Mandhani made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Carl Steinbach made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Carl Steinbach added a comment -

        Comments on phabricator.

        Show
        Carl Steinbach added a comment - Comments on phabricator.
        Hide
        Ashutosh Chauhan added a comment -

        Thanks Bhushan for considering the suggestion. I will take a look at your patch early next week.

        Show
        Ashutosh Chauhan added a comment - Thanks Bhushan for considering the suggestion. I will take a look at your patch early next week.
        Hide
        Bhushan Mandhani added a comment -

        Thanks Ashutosh. I've updated per your suggestion.

        Show
        Bhushan Mandhani added a comment - Thanks Ashutosh. I've updated per your suggestion.
        Bhushan Mandhani made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Bhushan Mandhani made changes -
        Field Original Value New Value
        Attachment HIVE-3400.1.patch.txt [ 12543345 ]
        Hide
        Ashutosh Chauhan added a comment -

        @Bhushan, It will be nice if this retrying logic is hidden from the users of HiveMetaStoreClient, otherwise they have to do

        else if (!metaStoreClient.isConnected()) {	
              metaStoreClient.connect();	
            }
        


        (as you have done in Hive.java) before every rpc call they make on a created HMSC, i.e., test whether client is connected and reconnect if its not. Best solution for this to proxy all calls on HMSC. You can find an example of how this is done in RetryingRawStore.java which proxies all calls to RawStore from HiveMetaStore precisely for this reason.

        Show
        Ashutosh Chauhan added a comment - @Bhushan, It will be nice if this retrying logic is hidden from the users of HiveMetaStoreClient , otherwise they have to do else if (!metaStoreClient.isConnected()) { metaStoreClient.connect(); } (as you have done in Hive.java) before every rpc call they make on a created HMSC, i.e., test whether client is connected and reconnect if its not. Best solution for this to proxy all calls on HMSC. You can find an example of how this is done in RetryingRawStore.java which proxies all calls to RawStore from HiveMetaStore precisely for this reason.
        Show
        Carl Steinbach added a comment - https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-Contributingyourwork
        Hide
        Carl Steinbach added a comment -

        Are you going to submit it for review?

        Show
        Carl Steinbach added a comment - Are you going to submit it for review?
        Hide
        Bhushan Mandhani added a comment -

        Yes, it is.

        Show
        Bhushan Mandhani added a comment - Yes, it is.
        Hide
        Carl Steinbach added a comment -

        Is this ready for review?

        Show
        Carl Steinbach added a comment - Is this ready for review?
        Hide
        Bhushan Mandhani added a comment -
        Show
        Bhushan Mandhani added a comment - Diff at https://reviews.facebook.net/D4791
        Bhushan Mandhani created issue -

          People

          • Assignee:
            Bhushan Mandhani
            Reporter:
            Bhushan Mandhani
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development