Uploaded image for project: 'Sentry'
  1. Sentry
  2. SENTRY-872 Uber jira for HMS HA + Sentry HA redesign
  3. SENTRY-1780

FullUpdateInitializer does not kill the threads whenever getFullHMSSnapshot throws an exception

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: Sentry
    • Labels:
      None

      Description

      Errors may happen inside the FullUpdateInitializer.getFullHMSSnapshot() causing to throw an exception. When
      this happens, the current threads running are not killed and continue consuming resources while they're alive.

      We should kill those threads. The information collected by them won't be used anymore.

      I noticed this issue on the logs:

      ### HMSFollower 1st run ######
      
      2017-05-23 13:57:56,976 (pool-5-thread-1) [DEBUG - org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:208)] HMSFollower run is called.        
      
      ### Starting to get a snapshot (2 databases found (db2, default) and submitted for execution) ###
      
      2017-05-23 13:57:56,978 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:396)] get full snapshot                          
      2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:401)] submit database: db2                           
      2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:401)] submit database: default                       
      
      ### Thread 1 started: db2 ###
      
      2017-05-23 13:57:56,979 (pool-150-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:346)] begin thread: db2                                         
      
      ### Waiting for db2 thread-1 ###
      
      2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:416)] Waiting for call java.util.concurrent.FutureTa$
      k@3b30d168                                                                                                                                                                                                  
      
      ### Thread 2 started: default ###
      
      2017-05-23 13:57:56,979 (pool-150-thread-2) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:346)] begin thread: default 
      
      ### Thread 1 failed ###                                    
      2017-05-23 13:57:57,007 (pool-150-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:212)] Failed to execute task on 1 attempt$
      . Sleeping for 1000 ms. Exception: org.apache.thrift.TApplicationException: get_database failed: out of sequence response                                                                                   
      2017-05-23 13:57:58,011 (pool-150-thread-1) [ERROR - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:232)] Task did not complete successfully $
      fter 1 tries
      
      ### Thread 1 failed, and throwing an exception from getFullHMSSnapshot() ###
      2017-05-23 13:57:58,011 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:418)] Call ended org.apache.sentry.hdfs.FullUpdateIn$
      tializer$CallResult@7cf642e8                                                                                                                                                                                
      2017-05-23 13:57:58,012 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:421)] throwing exception
      
      ### HMSFollower 2nd run ######
      
      2017-05-23 13:57:59,012 (pool-5-thread-1) [DEBUG - org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:208)] HMSFollower run is called.
      
      ### Thread 2 from 1st HMSFollower finished with a failure ####
      
      2017-05-23 13:58:14,722 (pool-150-thread-2) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:212)] Failed to execute task on 1 attempts
      . Sleeping for 1000 ms. Exception: org.apache.thrift.transport.TTransportException: java.net.SocketException: Socket closed
      2017-05-23 13:58:14,725 (pool-150-thread-2) [ERROR - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:232)] Task did not complete successfully a
      fter 2 tries
      

        Attachments

        1. SENTRY-1780.01-sentry-ha-redesign.patch
          7 kB
          Alex Kolbasov
        2. SENTRY-1780.01-sentry-ha-redesign.patch
          10 kB
          Alex Kolbasov

          Activity

            People

            • Assignee:
              akolb Alex Kolbasov
              Reporter:
              spena Sergio Peña
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: