Major Description
Errors may happen inside the FullUpdateInitializer.getFullHMSSnapshot() causing to throw an exception. When
this happens, the current threads running are not killed and continue consuming resources while they're alive.
We should kill those threads. The information collected by them won't be used anymore.
I noticed this issue on the logs:
### HMSFollower 1st run ###### 2017-05-23 13:57:56,976 (pool-5-thread-1) [DEBUG - org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:208)] HMSFollower run is called. ### Starting to get a snapshot (2 databases found (db2, default) and submitted for execution) ### 2017-05-23 13:57:56,978 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:396)] get full snapshot 2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:401)] submit database: db2 2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:401)] submit database: default ### Thread 1 started: db2 ### 2017-05-23 13:57:56,979 (pool-150-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:346)] begin thread: db2 ### Waiting for db2 thread-1 ### 2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:416)] Waiting for call java.util.concurrent.FutureTa$ k@3b30d168 ### Thread 2 started: default ### 2017-05-23 13:57:56,979 (pool-150-thread-2) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:346)] begin thread: default ### Thread 1 failed ### 2017-05-23 13:57:57,007 (pool-150-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:212)] Failed to execute task on 1 attempt$ . Sleeping for 1000 ms. Exception: org.apache.thrift.TApplicationException: get_database failed: out of sequence response 2017-05-23 13:57:58,011 (pool-150-thread-1) [ERROR - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:232)] Task did not complete successfully $ fter 1 tries ### Thread 1 failed, and throwing an exception from getFullHMSSnapshot() ### 2017-05-23 13:57:58,011 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:418)] Call ended org.apache.sentry.hdfs.FullUpdateIn$ tializer$CallResult@7cf642e8 2017-05-23 13:57:58,012 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:421)] throwing exception ### HMSFollower 2nd run ###### 2017-05-23 13:57:59,012 (pool-5-thread-1) [DEBUG - org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:208)] HMSFollower run is called. ### Thread 2 from 1st HMSFollower finished with a failure #### 2017-05-23 13:58:14,722 (pool-150-thread-2) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:212)] Failed to execute task on 1 attempts . Sleeping for 1000 ms. Exception: org.apache.thrift.transport.TTransportException: java.net.SocketException: Socket closed 2017-05-23 13:58:14,725 (pool-150-thread-2) [ERROR - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:232)] Task did not complete successfully a fter 2 tries