[HDDS-9030] Token can't be verified due to expired certificate 448729479270311

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Component/s: DN, OM

Environment

Ozone: 1.3.0

• Release Note: https://my.cloudera.com/knowledge/Ozone-Internal-SSL-certificate-expiration?id=358030
• Assignee: Unassigned
• Reporter: MLikeWater
• Due: 17/Jul/23
• Created: 17/Jul/23 08:36
• Updated: 17/Jul/23 09:33
• Resolved: 17/Jul/23 09:32

Description

When accessing data through ozone shell, the following problems occur:

$ ozone  fs -cat  /tgwarehouse/tgdw.db/test_2/part-00000-af5fcf66-941f-47ab-8de6-b1631d44dd05-c000
cat: Block token verification failed. Token can't be verified due to expired certificate 448729479270311

dn logs：

2023-07-17 16:29:45,351 [ChunkReader-0] INFO org.apache.hadoop.hdds.security.x509.certificate.client.DNCertificateClient: Getting certificate with certSerialId:448729479270311.
2023-07-17 16:29:47,388 [ChunkReader-0] INFO org.apache.hadoop.ozone.container.common.impl.HddsDispatcher: Operation: GetBlock , Trace ID:  , Message: Block token verification failed. Token can't be verified due to expired certificate 448729479270311 , Result: BLOCK_TOKEN_VERIFICATION_FAILED , StorageContainerException Occurred.
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Block token verification failed. Token can't be verified due to expired certificate 448729479270311
    at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:213)
    at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.lambda$dispatch$0(HddsDispatcher.java:170)
    at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
    at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:169)
    at org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:57)
    at org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:50)
    at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
    at org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
    at org.apache.hadoop.hdds.tracing.GrpcServerInterceptor$1.onMessage(GrpcServerInterceptor.java:49)
    at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:332)
    at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:315)
    at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:834)
    at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hdds.security.token.BlockTokenException: Token can't be verified due to expired certificate 448729479270311
    at org.apache.hadoop.hdds.security.token.ShortLivedTokenVerifier.verify(ShortLivedTokenVerifier.java:105)
    at org.apache.hadoop.hdds.security.token.CompositeTokenVerifier.verify(CompositeTokenVerifier.java:43)
    at org.apache.hadoop.hdds.security.token.TokenVerifier.verify(TokenVerifier.java:71)
    at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateToken(HddsDispatcher.java:453)
    at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:210)
    ... 16 more

When view certs, the om and dn certificates have expired:

ozone admin cert list