[HDDS-7930] input stream does not refresh expired block token

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Labels: pull-request-available
• Target Version/s: 1.4.0, 1.3.1
• Assignee: Wei-Chiu Chuang
• Reporter: Wei-Chiu Chuang
• Created: 08/Feb/23 17:51
• Updated: 07/Mar/24 07:15
• Resolved: 15/Mar/23 05:08

Description

On a Kerberized cluster HBase ycsb client doesn'twork after the cluster is up for more than 1 day. The RegionServer has the following error which suggests it was due to expired block token used by Ozone input stream.

 

2023-02-08 15:49:51,464 ERROR org.apache.hadoop.hdds.scm.XceiverClientGrpc: Failed to execute command ReadChunk on the pipeline Pipeline[ Id: 39993807-ab46-4996-9313-c397e42dc172, Nodes
: f244dd2f-678d-4f02-b4c7-021ea7c3a5e5(weichiu-2.weichiu.root.hwx.site/172.27.161.6)f7a94508-4f6e-401f-a50d-8f05570de76a(weichiu-3.weichiu.root.hwx.site/172.27.128.9)8c7429af-315e-49d8-
8a64-42ed2b39cb19(weichiu-1.weichiu.root.hwx.site/172.27.74.76), ReplicationConfig: STANDALONE/THREE, State:OPEN, leaderId:8c7429af-315e-49d8-8a64-42ed2b39cb19, CreationTimestamp2023-02
-07T05:36:46.179Z[UTC]].
2023-02-08 15:49:51,465 INFO org.apache.hadoop.hdds.scm.storage.BlockInputStream: Unable to read information for block conID: 4 locID: 111677748019204244 bcsId: 754837 from pipeline Pip
elineID=39993807-ab46-4996-9313-c397e42dc172: Block token verification failed. Expired token for user: hbase (auth:SIMPLE)
2023-02-08 15:49:51,474 INFO org.apache.hadoop.hdds.scm.storage.BlockInputStream: Unable to read information for block conID: 4 locID: 111677748019204244 bcsId: 754837 from pipeline Pip
elineID=39993807-ab46-4996-9313-c397e42dc172: Block token verification failed. Expired token for user: hbase (auth:SIMPLE)
2023-02-08 15:49:51,483 INFO org.apache.hadoop.hdds.scm.storage.BlockInputStream: Unable to read information for block conID: 1 locID: 111677748019204242 bcsId: 720383 from pipeline Pip
elineID=ef52af00-6469-4acf-a7ff-b63a98295ba2: Block token verification failed. Expired token for user: hbase (auth:SIMPLE)
2023-02-08 15:49:51,486 INFO org.apache.hadoop.hdds.scm.storage.BlockInputStream: Unable to read information for block conID: 2 locID: 111677748019204246 bcsId: 720467 from pipeline Pip
elineID=ef52af00-6469-4acf-a7ff-b63a98295ba2: Block token verification failed. Expired token for user: hbase (auth:SIMPLE)
2023-02-08 15:49:51,499 ERROR org.apache.hadoop.hbase.regionserver.StoreFileReader: Bad bloom filter data -- proceeding without
java.lang.IllegalArgumentException: Failed to load Bloom block
        at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.getBloomBlock(CompoundBloomFilter.java:118)
        at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:94)
        at org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:416)
        at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:309)
        at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:242)
        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:479)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:463)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:245)
        at org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:1766)
        at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1757)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:7053)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:7033)
        at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:3036)
        at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:3016)
        at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2998)
        at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2992)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2624)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2548)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45815)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:384)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:138)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:371)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:351)
Caused by: org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Block token verification failed. Expired token for user: hbase (auth:SIMPLE)
        at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:631)
        at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.lambda$getValidatorList$0(ContainerProtocolCalls.java:638)
        at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientSpi.java:139)
        at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:291)
        at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:431)
        at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkDataIntoBuffers(ChunkInputStream.java:404)
        at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:389)
        at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:321)
        at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:175)
        at org.apache.hadoop.hdds.scm.storage.ByteArrayReader.readFromBlock(ByteArrayReader.java:57)
        at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readWithStrategy(BlockInputStream.java:303)
        at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:54)
        at org.apache.hadoop.hdds.scm.storage.ByteArrayReader.readFromBlock(ByteArrayReader.java:57)
        at org.apache.hadoop.hdds.scm.storage.MultipartInputStream.readWithStrategy(MultipartInputStream.java:96)
        at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:54)
        at org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:64)
        at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:78)
        at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:97)
        at org.apache.hadoop.hbase.io.util.BlockIOUtils.preadWithExtra(BlockIOUtils.java:233)
        at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1456)
        at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1679)
        at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1490)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1310)
        at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.getBloomBlock(CompoundBloomFilter.java:114)
        ... 22 more 

I'm not sure if it's a bug in HBase or Ozone. File as a HDDS bug for now.

Issue Links

• is related to: HDDS-9734 ChunkInputStream should use new token after pipeline refresh
• links to: GitHub Pull Request #4378