[HDDS-10632] [Hbase Ozone] HMaster aborted with "IOException: Inconsistent read"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: HDDS-7593
• Component/s: SCM
• Labels: pull-request-available
• Assignee: Ashish Kumar
• Reporter: Pratyush Bhatt
• Created: 02/Apr/24 04:56
• Updated: 14/Jun/24 19:50
• Resolved: 14/Jun/24 19:50

Description

Both the HMasters are down, the HMaster fails with:

2024-04-01 13:15:51,517 ERROR org.apache.hadoop.hbase.master.HMaster: Failed to become active master
java.io.IOException: Inconsistent read for blockID=conID: 8366 locID: 113750153625964072 bcsId: 0 length=268435456 position=83 numBytesToRead=1 numBytesRead=-1
        at org.apache.hadoop.ozone.client.io.KeyInputStream.checkPartBytesRead(KeyInputStream.java:191)
        at org.apache.hadoop.hdds.scm.storage.MultipartInputStream.readWithStrategy(MultipartInputStream.java:97)
        at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:56)
        at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:43)
        at org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:55)
        at java.io.FilterInputStream.read(FilterInputStream.java:83)
        at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.parseDelimitedFrom(ProtobufUtil.java:3576)
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:348)
        at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:95)
        at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:83)
        at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:5298)
        at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:5182)
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:998)
        at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:939)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7903)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7860)
        at org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:307)
        at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:424)
        at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:122)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:848)
        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2216)
        at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528)
        at java.lang.Thread.run(Thread.java:748)
2024-04-01 13:15:51,517 ERROR org.apache.hadoop.hbase.master.HMaster: ***** ABORTING master vc0121.halxg.cloudera.com,22001,1711989581483: Unhandled exception. Starting shutdown. *****
java.io.IOException: Inconsistent read for blockID=conID: 8366 locID: 113750153625964072 bcsId: 0 length=268435456 position=83 numBytesToRead=1 numBytesRead=-1
        at org.apache.hadoop.ozone.client.io.KeyInputStream.checkPartBytesRead(KeyInputStream.java:191)
        at org.apache.hadoop.hdds.scm.storage.MultipartInputStream.readWithStrategy(MultipartInputStream.java:97)
        at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:56)
        at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:43)
        at org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:55)
        at java.io.FilterInputStream.read(FilterInputStream.java:83)
        at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.parseDelimitedFrom(ProtobufUtil.java:3576)
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:348)
        at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:95)
        at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:83)
        at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:5298)
        at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:5182)
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:998)
        at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:939)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7903)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7860)
        at org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:307)
        at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:424)
        at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:122)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:848)
        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2216)
        at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528)
        at java.lang.Thread.run(Thread.java:748)
2024-04-01 13:15:51,517 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: ***** STOPPING region server 'vc0121.xyz,22001,1711989581483' ***** 

Issue Links

• is related to: HDDS-8942 Intermittent failure in ITestOzoneContractCreate#testSyncable
• links to: GitHub Pull Request #6810