[HDDS-4644] Block token verification failed: no READ permission for WriteChunk

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.1.0
• Component/s: Security
• Labels: pull-request-available
• Target Version/s: 1.1.0
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 05/Jan/21 12:30
• Updated: 06/Jan/21 21:14
• Resolved: 05/Jan/21 22:29

Description

With HDDS-4558 committed, secure acceptance test logs increased considerably (over 1GB).

https://github.com/apache/ozone/actions/runs/462095579

I think the root cause is that WriteChunk request may need to also ReadChunk, but now it fails because it only has write access:

datanode_3  | 2021-01-05 10:41:23,067 [ChunkWriter-1-0] INFO impl.HddsDispatcher: Operation: ReadChunk , Trace ID:  , Message: Block token verification failed. Block token with conID: 1 locID: 105502689303461889 doesn't have READ permission , Result: BLOCK_TOKEN_VERIFICATION_FAILED , StorageContainerException Occurred.
datanode_3  | org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Block token verification failed. Block token with conID: 1 locID: 105502689303461889 doesn't have READ permission
datanode_3  | 	at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:214)
datanode_3  | 	at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.lambda$dispatch$0(HddsDispatcher.java:171)
datanode_3  | 	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
datanode_3  | 	at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:170)
datanode_3  | 	at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:398)
datanode_3  | 	at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.readStateMachineData(ContainerStateMachine.java:585)
datanode_3  | 	at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$read$5(ContainerStateMachine.java:656)
datanode_3  | 	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
datanode_3  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
datanode_3  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
datanode_3  | 	at java.base/java.lang.Thread.run(Thread.java:834)
datanode_3  | Caused by: org.apache.hadoop.hdds.security.token.BlockTokenException: Block token with conID: 1 locID: 105502689303461889 doesn't have READ permission
datanode_3  | 	at org.apache.hadoop.hdds.security.token.BlockTokenVerifier.verify(BlockTokenVerifier.java:131)
datanode_3  | 	at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateBlockToken(HddsDispatcher.java:431)
datanode_3  | 	at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:211)
datanode_3  | 	... 10 more
datanode_3  | 2021-01-05 10:41:23,083 [ChunkWriter-1-0] ERROR ratis.ContainerStateMachine: gid group-5BCDF056E270 : ReadStateMachine failed. cmd ReadChunk logIndex 4 msg : Block token verification failed. Block token with conID: 1 locID: 105502689303461889 doesn't have READ permission Container Result: BLOCK_TOKEN_VERIFICATION_FAILED

CC elek xyao

Issue Links

• is caused by: HDDS-4558 Support Ozone block token with access mode check
• links to: GitHub Pull Request #1758