Uploaded image for project: 'Sentry (Retired)'
  1. Sentry (Retired)
  2. SENTRY-872 Uber jira for HMS HA + Sentry HA redesign
  3. SENTRY-1780

FullUpdateInitializer does not kill the threads whenever getFullHMSSnapshot throws an exception

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0
    • 2.0.0
    • Sentry
    • None

    Description

      Errors may happen inside the FullUpdateInitializer.getFullHMSSnapshot() causing to throw an exception. When
      this happens, the current threads running are not killed and continue consuming resources while they're alive.

      We should kill those threads. The information collected by them won't be used anymore.

      I noticed this issue on the logs:

      ### HMSFollower 1st run ######
      
      2017-05-23 13:57:56,976 (pool-5-thread-1) [DEBUG - org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:208)] HMSFollower run is called.        
      
      ### Starting to get a snapshot (2 databases found (db2, default) and submitted for execution) ###
      
      2017-05-23 13:57:56,978 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:396)] get full snapshot                          
      2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:401)] submit database: db2                           
      2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:401)] submit database: default                       
      
      ### Thread 1 started: db2 ###
      
      2017-05-23 13:57:56,979 (pool-150-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:346)] begin thread: db2                                         
      
      ### Waiting for db2 thread-1 ###
      
      2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:416)] Waiting for call java.util.concurrent.FutureTa$
      k@3b30d168                                                                                                                                                                                                  
      
      ### Thread 2 started: default ###
      
      2017-05-23 13:57:56,979 (pool-150-thread-2) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:346)] begin thread: default 
      
      ### Thread 1 failed ###                                    
      2017-05-23 13:57:57,007 (pool-150-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:212)] Failed to execute task on 1 attempt$
      . Sleeping for 1000 ms. Exception: org.apache.thrift.TApplicationException: get_database failed: out of sequence response                                                                                   
      2017-05-23 13:57:58,011 (pool-150-thread-1) [ERROR - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:232)] Task did not complete successfully $
      fter 1 tries
      
      ### Thread 1 failed, and throwing an exception from getFullHMSSnapshot() ###
      2017-05-23 13:57:58,011 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:418)] Call ended org.apache.sentry.hdfs.FullUpdateIn$
      tializer$CallResult@7cf642e8                                                                                                                                                                                
      2017-05-23 13:57:58,012 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:421)] throwing exception
      
      ### HMSFollower 2nd run ######
      
      2017-05-23 13:57:59,012 (pool-5-thread-1) [DEBUG - org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:208)] HMSFollower run is called.
      
      ### Thread 2 from 1st HMSFollower finished with a failure ####
      
      2017-05-23 13:58:14,722 (pool-150-thread-2) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:212)] Failed to execute task on 1 attempts
      . Sleeping for 1000 ms. Exception: org.apache.thrift.transport.TTransportException: java.net.SocketException: Socket closed
      2017-05-23 13:58:14,725 (pool-150-thread-2) [ERROR - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:232)] Task did not complete successfully a
      fter 2 tries
      

      Attachments

        1. SENTRY-1780.01-sentry-ha-redesign.patch
          7 kB
          Alex Kolbasov
        2. SENTRY-1780.01-sentry-ha-redesign.patch
          10 kB
          Alex Kolbasov

        Activity

          People

            akolb Alex Kolbasov
            spena Sergio Peña
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: