Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • Hive Binding
    • None

    Description

      Looking at the existing logic of HMSFollower, upon the connection loss to HMS the client never attempts to reconnect to HMS but rather just errors out with "Broken pipe" indefinitely.

      2017-02-02 20:00:03,314 ERROR org.apache.sentry.service.thrift.HMSFollower: ThriftException occured fetching Notification entries, will try
      org.apache.thrift.transport.TTransportException
      	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
      	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
      	at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
      	at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
      	at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
      	at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
      	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
      	at org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
      	at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
      	at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
      	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
      	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
      	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_current_notificationEventId(ThriftHiveMetastore.java:4233)
      	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_current_notificationEventId(ThriftHiveMetastore.java:4221)
      	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getCurrentNotificationEventId(HiveMetaStoreClient.java:2030)
      	at org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:232)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      2017-02-02 20:00:03,317 ERROR org.apache.sentry.service.thrift.HMSFollower: Encountered Socket Exception during fetching Notification entries, will reconnect to HMS
      org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
      	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
      	at org.apache.thrift.transport.TSaslTransport.flush(TSaslTransport.java:501)
      	at org.apache.thrift.transport.TSaslClientTransport.flush(TSaslClientTransport.java:37)
      	at org.apache.hadoop.hive.thrift.TFilterTransport.flush(TFilterTransport.java:77)
      	at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
      	at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
      	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_current_notificationEventId(ThriftHiveMetastore.java:4227)
      	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_current_notificationEventId(ThriftHiveMetastore.java:4220)
      	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getCurrentNotificationEventId(HiveMetaStoreClient.java:2030)
      	at org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:227)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.net.SocketException: Broken pipe
      	at java.net.SocketOutputStream.socketWrite0(Native Method)
      	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
      	at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
      	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
      	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
      	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
      	... 16 more
      

      To improve this, we can extend HMSFollower to scan for SocketException errors and if received, it can instead reconnect to HMS.

      Attachments

        1. SENTRY-1621.01-sentry-ha-redesign.patch
          6 kB
          Vamsee K. Yarlagadda

        Issue Links

          Activity

            People

              vamsee Vamsee K. Yarlagadda
              vamsee Vamsee K. Yarlagadda
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Slack

                  Issue deployment