[HDDS-11239] Flakiness in KeyOutputStream exception handling

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Labels: pull-request-available
• Assignee: Duong
• Reporter: Duong
• Created: 26/Jul/24 17:10
• Updated: 21/Aug/24 22:20
• Resolved: 21/Aug/24 20:33

Description

2024-07-23 05:32:26,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplit: Compaction failed region=11loadtesttabtyui22aaqw23,a22221f2,1721725742844.9bcd68da6a7f7c0150854b20b2340a41., storeName=9bcd68da6a7f7c0150854b20b2340a41/ch8, priority=13, startTime=1721737880870
java.io.IOException: java.lang.IllegalArgumentException
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.ensureIOException(AbstractFSWAL.java:858)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:853)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:809)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:858)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:847)
        at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullMarkerAppendTransaction(WALUtil.java:163)
        at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:140)
        at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeCompactionMarker(WALUtil.java:74)
        at org.apache.hadoop.hbase.regionserver.HStore.writeCompactionWalRecord(HStore.java:1262)
        at org.apache.hadoop.hbase.regionserver.HStore.lambda$replaceStoreFiles$3(HStore.java:1272)
        at org.apache.hadoop.hbase.regionserver.StoreEngine.replaceStoreFiles(StoreEngine.java:483)
        at org.apache.hadoop.hbase.regionserver.HStore.replaceStoreFiles(HStore.java:1270)
        at org.apache.hadoop.hbase.regionserver.HStore.doCompaction(HStore.java:1206)
        at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1188)
        at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2261)
        at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:625)
        at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:673)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException
        at org.apache.hadoop.ozone.shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:129)
        at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:318)
        at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleFlushOrClose(KeyOutputStream.java:499)
        at org.apache.hadoop.ozone.client.io.KeyOutputStream.hsync(KeyOutputStream.java:463)
        at org.apache.hadoop.ozone.client.io.OzoneOutputStream.hsync(OzoneOutputStream.java:118)
        at org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:184)
        at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:149)
        at org.apache.hadoop.fs.ozone.OzoneFSOutputStream.hsync(OzoneFSOutputStream.java:80)
        at org.apache.hadoop.fs.ozone.OzoneFSOutputStream.hflush(OzoneFSOutputStream.java:75)
        at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:136)
        at org.apache.h

Issue Links

• fixes: HDDS-11328 ContainerStateMachine should not crash because of CHUNK_FILE_INCONSISTENCY
• is duplicated by: HDDS-11269 KeyOutputStream throws "java.io.IOException: : Stream is closed"
• links to: GitHub Pull Request #7047