[HIVE-13090] Hive metastore crashes on NPE with ZooKeeperTokenStore

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0, 1.1.1, 1.2.1
• Fix Version/s: 1.3.0, 2.0.1, 2.1.0
• Component/s: Metastore, Security
• Assignee: Piotr Wikieł
• Reporter: Thejas Nair
• Created: 18/Feb/16 21:58
• Updated: 05/Sep/17 09:58
• Resolved: 22/Feb/16 17:11

Description

Observed that hive metastore shutdown with NPE from ZookeeperTokenStore.

INFO  [pool-5-thread-192]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(714)) - 191: Metastore shutdown complete.
 INFO  [pool-5-thread-192]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(340)) - ugi=cvdpqap	ip=/19.1.2.129	cmd=Metastore shutdown complete.	
 ERROR [Thread[Thread-6,5,main]]: thrift.TokenStoreDelegationTokenSecretManager (TokenStoreDelegationTokenSecretManager.java:run(331)) - ExpiredTokenRemover thread received unexpected exception. org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Failed to decode token
org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Failed to decode token
	at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.getToken(ZooKeeperTokenStore.java:401)
	at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.removeExpiredTokens(TokenStoreDelegationTokenSecretManager.java:256)
	at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager$ExpiredTokenRemover.run(TokenStoreDelegationTokenSecretManager.java:319)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NullPointerException
	at java.io.ByteArrayInputStream.<init>(ByteArrayInputStream.java:106)
	at org.apache.hadoop.security.token.delegation.HiveDelegationTokenSupport.decodeDelegationTokenInformation(HiveDelegationTokenSupport.java:53)
	at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.getToken(ZooKeeperTokenStore.java:399)
	... 3 more
 INFO  [Thread-3]: metastore.HiveMetaStore (HiveMetaStore.java:run(5639)) - Shutting down hive metastore.

Issue Links

• breaks: HIVE-15090 Temporary DB failure can stop ExpiredTokenRemover thread
• duplicates: HIVE-13075 Metastore shuts down when no delegation token is found in ZooKeeper