[HDDS-9020] Datanodes fails to start up when secret key has not yet been initialized in SCM.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Component/s: OM, Ozone Datanode
• Labels: pull-request-available
• Assignee: Duong
• Reporter: Duong
• Created: 14/Jul/23 17:56
• Updated: 17/Jul/23 18:07
• Resolved: 17/Jul/23 18:07

Description

When a datanode/OM starts up while SCM has not finished initializing secret keys yet, the startup fails because Datanode/OM needs to prefetch the current active secret keys.

2023-07-13 16:06:22,369 [main] ERROR org.apache.hadoop.ozone.HddsDatanodeService: Exception in HddsDatanodeService.
java.lang.RuntimeException: Can't start the HDDS datanode plugin
	at org.apache.hadoop.ozone.HddsDatanodeService.start(HddsDatanodeService.java:361)
	at org.apache.hadoop.ozone.HddsDatanodeService.start(HddsDatanodeService.java:235)
	at org.apache.hadoop.ozone.HddsDatanodeService.call(HddsDatanodeService.java:203)
	at org.apache.hadoop.ozone.HddsDatanodeService.call(HddsDatanodeService.java:93)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine.execute(CommandLine.java:2078)
	at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
	at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
	at org.apache.hadoop.ozone.HddsDatanodeService.main(HddsDatanodeService.java:185)
Caused by: org.apache.hadoop.hdds.security.exception.SCMSecretKeyException: Secret key initialization is not finished yet.
	at org.apache.hadoop.hdds.protocolPB.SecretKeyProtocolClientSideTranslatorPB.handleError(SecretKeyProtocolClientSideTranslatorPB.java:101)
	at org.apache.hadoop.hdds.protocolPB.SecretKeyProtocolClientSideTranslatorPB.submitRequest(SecretKeyProtocolClientSideTranslatorPB.java:89)
	at org.apache.hadoop.hdds.protocolPB.SecretKeyProtocolClientSideTranslatorPB.getCurrentSecretKey(SecretKeyProtocolClientSideTranslatorPB.java:127)
	at org.apache.hadoop.hdds.security.symmetric.DefaultSecretKeySignerClient.start(DefaultSecretKeySignerClient.java:72)
	at org.apache.hadoop.hdds.security.symmetric.DefaultSecretKeyClient.start(DefaultSecretKeyClient.java:50)
	at org.apache.hadoop.ozone.HddsDatanodeService.start(HddsDatanodeService.java:312)
	... 13 more
2023-07-13 16:06:22,382 [shutdown-hook-0] INFO org.apache.hadoop.ozone.HddsDatanodeService: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down HddsDatanodeService at ....
************************************************************/  

Issue Links

• links to: GitHub Pull Request #5068