Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-24605

Metastore link leak in hiveserver2

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.1.0
    • None
    • HiveServer2
    • None

    Description

      When we use hiveserver and connect to metastore server, we face metastore link leak as below:

       

      2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - db.metaStoreClient.isSameConfObj(c) is false
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - db.metaStoreClient.isCompatibleWith(c) is false
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - hive db is org.apache.hadoop.hive.ql.metadata.Hive@ec83e59
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - METASTORE_FILTER_HOOK:org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - db.metaStoreClient.isSameConfObj(c) is false
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - db.metaStoreClient.isCompatibleWith(c) is false
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2021-01-08T11:10:00,196 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - Closing current thread's connection to Hive Metastore.
2021-01-08T11:10:00,201 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Closed a connection to metastore, current connections: 0
2021-01-08T11:10:00,201 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - Closing current thread's connection to Hive Metastore.
2021-01-08T11:10:00,201 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - current Hive is org.apache.hadoop.hive.ql.metadata.Hive@5898c9e7
2021-01-08T11:10:00,202 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Trying to connect to metastore with URI thrift://xxx
2021-01-08T11:10:00,206 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Opened a connection to metastore, current connections: 1
2021-01-08T11:10:00,206 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Connected to metastore.
2021-01-08T11:10:00,215 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: parse.SemanticAnalyzer (:()) - Completed phase 1 of Semantic Analysis
2021-01-08T11:10:00,215 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: parse.SemanticAnalyzer (:()) - Get metadata for source tables
2021-01-08T11:10:00,215 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: parse.SemanticAnalyzer (:()) - current db in getMetaData is org.apache.hadoop.hive.ql.metadata.Hive@Hive@ec83e59
2021-01-08T11:10:00,215 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - current Hive is org.apache.hadoop.hive.ql.metadata.Hive@5898c9e7
2021-01-08T11:10:00,215 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Trying to connect to metastore with URI thrift://xxx
2021-01-08T11:10:00,218 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Opened a connection to metastore, current connections: 2
2021-01-08T11:10:00,219 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Connected to metastore.
 
.....
 
2021-01-08T11:12:16,119 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - metadata.Hive (:()) - close current: org.apache.hadoop.hive.ql.metadata.Hive@5898c9e7
2021-01-08T11:12:16,121 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: metadata.Hive (:()) - Closing current thread's connection to Hive Metastore, current is org.apache.hadoop.hive.ql.metadata.Hive@5898c9e7
2021-01-08T11:12:16,121 INFO  [qid=61_142_176_249_26122_20210108110957686_0]: hive.metastore (:()) - Closed a connection to metastore, current connections: 1

       

       

      because HIVE_AUTHORIZATION_MANAGER is set to org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory

      when in SessionState.setAuthorizerV2Config, sessionConf is changed, so Hive.create will  change and replace the old one in Hive.hiveDB. But the old Hive instance is already assigned in BaseSemanticAnalyzer.db, and when it's called, metaStoreClient will be assigned to a new one and create a new metastore link.  The metastoreClient in new Hive instance can be closed, but the old one assigned in BaseSemanticAnalyzer  will not close when we call Hive.close and Hive.closeCurrent, which may lead to a link leak.

      So, can we use Hive.get() instead of using db.xxx in BaseSemanticAnalyzer, or it's better to re-assign to the new Hive instance in BaseSemanticAnalyzer  instead of  using the old one.

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            snodawn snodawn
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: