[HDFS-16591] StateStoreZooKeeper fails to initialize

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.4.0, 3.3.5
• Component/s: rbf
• Labels: pull-request-available
• Assignee: Hector Sandoval Chaverri
• Reporter: Hector Sandoval Chaverri
• Created: 23/May/22 16:57
• Updated: 29/Jun/22 00:06
• Resolved: 29/Jun/22 00:06

Description

MembershipStore and MountTableStore are failing to initialize, logging the following errors on the Router logs:

2022-05-23 16:43:01,156 ERROR org.apache.hadoop.hdfs.server.federation.router.RouterHeartbeatService: Cannot get version for class org.apache.hadoop.hdfs.server.federation.store.MembershipStore
org.apache.hadoop.hdfs.server.federation.store.StateStoreUnavailableException: Cached State Store not initialized, MembershipState records not valid
	at org.apache.hadoop.hdfs.server.federation.store.CachedRecordStore.checkCacheAvailable(CachedRecordStore.java:106)
	at org.apache.hadoop.hdfs.server.federation.store.CachedRecordStore.getCachedRecords(CachedRecordStore.java:227)
	at org.apache.hadoop.hdfs.server.federation.router.RouterHeartbeatService.getStateStoreVersion(RouterHeartbeatService.java:131)
	at org.apache.hadoop.hdfs.server.federation.router.RouterHeartbeatService.updateStateStore(RouterHeartbeatService.java:92)
	at org.apache.hadoop.hdfs.server.federation.router.RouterHeartbeatService.periodicInvoke(RouterHeartbeatService.java:159)
	at org.apache.hadoop.hdfs.server.federation.router.PeriodicService$1.run(PeriodicService.java:178)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

After investigating, we noticed that ZKDelegationTokenSecretManager normally initializes properties for ZooKeeper clients to connect using SASL/Kerberos. If ZKDelegationTokenSecretManager is replaced with a new SecretManager, the SASL properties don't get configured and any StateStores that connect to ZooKeeper fail with the above error. 

 A potential way to fix this is by setting the JaasConfiguration (currently done in ZKDelegationTokenSecretManager) as part of the StateStoreZooKeeperImpl initialization method.

Issue Links

• links to: GitHub Pull Request #4447