[HBASE-25875] RegionServer failed to start due to IllegalThreadStateException in AuthenticationTokenSecretManager.start

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.4
• Hadoop Flags: Reviewed
• Assignee: Pankaj Kumar
• Reporter: Pankaj Kumar
• Created: 10/May/21 11:34
• Updated: 26/May/21 15:12
• Resolved: 25/May/21 18:49

Description

RegionServer failed to complete initialization and aborted during AuthenticationTokenSecretManager#leaderElector start.

Observed following WARN log,

2021-05-03 07:59:01,848 | WARN  | RS-EventLoopGroup-1-6 | Thread leaderElector[ZKSecretWatcher-leaderElector:56] is stopped or not alive | org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.retrievePassword(AuthenticationTokenSecretManager.java:153)
2021-05-03 07:59:01,848 | INFO  | RS-EventLoopGroup-1-6 | Thread leaderElector [ZKSecretWatcher-leaderElector:56] is started | org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.retrievePassword(AuthenticationTokenSecretManager.java:156)
2021-05-03 07:59:01,854 | INFO  | ZKSecretWatcher-leaderElector | Found existing leader with ID: RS-IP-PORT-StartCode | org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.waitToBecomeLeader(ZKLeaderManager.java:130)

As per the code, AuthenticationTokenSecretManager#leaderElector is started while retrieving password before AuthenticationTokenSecretManager#start,
https://github.com/apache/hbase/blob/8c2332d46532135723cc7a6084a2a125f3d9d8db/hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java#L155

So IllegalThreadStateException occured during AuthenticationTokenSecretManager#start,
https://github.com/apache/hbase/blob/8c2332d46532135723cc7a6084a2a125f3d9d8db/hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java#L107

2021-05-03 07:59:02,066 | ERROR | main | Failed construction RegionServer | org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:775)
java.lang.IllegalThreadStateException
	at java.lang.Thread.start(Thread.java:708)
	at org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.start(AuthenticationTokenSecretManager.java:107)
	at org.apache.hadoop.hbase.ipc.NettyRpcServer.start(NettyRpcServer.java:131)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.start(RSRpcServices.java:1695)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:756)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:3270)
	at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:63)
	at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)

Issue Links

• links to: GitHub Pull Request #3250
• links to: GitHub Pull Request #3272