XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: Hdfs Plugin
    • Labels:
      None

      Description

      When HMSFollower exits in an abnormal way, for all the following connection would encounter out of sequence and SocketTimeoutException: Read timed out. Looking at HIVE-6893, it seems to relate to leakage connection problem.

      2017-02-15 19:03:42,822 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete successfully after 0 tries. Exception got: org.apache.thrift.TApplicationException: get_database failed: out of sequence response
      2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete successfully after 0 tries. Exception got: MetaException(message:Got exception: org.apache.thrift.transport.TTransportException null)
      2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to metastore, current connections: 0
      2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: Exception occurred persisting Hive full snapshot into DB
      java.lang.RuntimeException: org.apache.thrift.TApplicationException: get_database failed: out of sequence response
              at org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324)
              at org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343)
              at org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.thrift.TApplicationException: get_database failed: out of sequence response
              at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649)
              at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213)
              at org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256)
              at org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110)
              at org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150)
              at org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79)
              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
              ... 3 more
      2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: Making a kerberos connection to HMS

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                akolb Alex Kolbasov
                Reporter:
                hahao Hao Hao
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: