Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-4964

Drill fails to connect to hive metastore after hive metastore is restarted unless drillbits are restarted also

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.8.0
    • Fix Version/s: 1.9.0
    • Component/s: None
    • Labels:
      None

      Description

      It is found that if Hive Metastore is restarted then Drillbit also needs to be restarted to further query and connect to hive metastore.

      Repro Steps:
      ===========
      1) Start HiveMetastore and drillbit.
      2) Start Drillbit client with Scheman as hive and run a simple query like "show databases"
      Command to start client: sqlline -u "jdbc:drill:schema=hive;drillbit=<name>"
      3) restart hive metastore
      4) Execute same query "show databases" on existing drillclient or new one. You will see that hive default database is not listed. If you query any hive data then it will fail.

      Log snippet from drillbit.log:
      ======================

      2016-10-25 18:32:00,561 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 27eff86e-e8fb-3d91-eb88-4af75ff6d174: show databases
      2016-10-25 18:32:00,563 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] DEBUG o.a.d.e.s.h.HBaseStoragePluginConfig - Initializing HBase StoragePlugin configuration with zookeeper quorum 'localhost', port '2181'.
      2016-10-25 18:32:00,595 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] WARN o.a.d.e.s.h.schema.HiveSchemaFactory - Failure while attempting to access HiveDatabase 'default'.
      java.util.concurrent.ExecutionException: MetaException(message:Got exception: org.apache.thrift.transport.TTransportException null)
      at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na]
      at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-18.0.jar:na]
      at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na]
      at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137) ~[guava-18.0.jar:na]
      at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348) ~[guava-18.0.jar:na]
      at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2320) ~[guava-18.0.jar:na]
      at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
      at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
      at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
      at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
      at org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.getDatabases(DrillHiveMetaStoreClient.java:415) ~[drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
      at org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getSubSchema(HiveSchemaFactory.java:139) [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
      at org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.<init>(HiveSchemaFactory.java:133) [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
      at org.apache.drill.exec.store.hive.schema.HiveSchemaFactory.registerSchemas(HiveSchemaFactory.java:118) [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
      at org.apache.drill.exec.store.hive.HiveStoragePlugin.registerSchemas(HiveStoragePlugin.java:100) [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
      at org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365) [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
      at org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72) [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
      at org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:61) [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
      at org.apache.drill.exec.ops.QueryContext.getRootSchema(QueryContext.java:147) [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
      at org.apache.drill.exec.ops.QueryContext.getRootSchema(QueryContext.java:137) [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
      at org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema(QueryContext.java:123) [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
      at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:62) [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
      at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1008) [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
      at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:264) [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_111]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_111]
      at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
      Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Got exception: org.apache.thrift.transport.TTransportException null
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1213) ~[hive-metastore-1.2.0-mapr-1608.jar:1.2.0-mapr-1608]

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                shamirwasia Sorabh Hamirwasia
                Reporter:
                shamirwasia Sorabh Hamirwasia
                Reviewer:
                Krystal
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: