[HDDS-10651] HBase WAL splitting fails due to lease recovery

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Not A Bug
• Assignee: Sammi Chen
• Reporter: Wei-Chiu Chuang
• Created: 05/Apr/24 01:35
• Updated: 29/Apr/24 18:26
• Resolved: 29/Apr/24 18:26

Description

HBase RS log:

2024-04-04 21:33:03,337 INFO org.apache.hadoop.hbase.wal.WALSplitter: Splitting ofs://ozone1710266930/vol1/bucket1/hbase/WALs/ccycloud-2.weichiu-src.root.comops.site,22101,1712266104607-spli
tting/ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266120442, size=83 (83bytes)
2024-04-04 21:33:03,337 INFO org.apache.hadoop.hbase.wal.WALSplitter: Splitting ofs://ozone1710266930/vol1/bucket1/hbase/WALs/ccycloud-2.weichiu-src.root.comops.site,22101,1712266104607-spli
tting/ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266382475, size=83 (83bytes)
2024-04-04 21:33:03,340 INFO org.apache.hadoop.hbase.util.RecoverLeaseFSUtils: Recover lease on dfs file ofs://ozone1710266930/vol1/bucket1/hbase/WALs/ccycloud-2.weichiu-src.root.comops.site
,22101,1712266104607-splitting/ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266120442
2024-04-04 21:33:03,340 INFO org.apache.hadoop.hbase.util.RecoverLeaseFSUtils: Recover lease on dfs file ofs://ozone1710266930/vol1/bucket1/hbase/WALs/ccycloud-2.weichiu-src.root.comops.site
,22101,1712266104607-splitting/ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266382475
2024-04-04 21:33:03,361 WARN org.apache.hadoop.hbase.util.RecoverLeaseFSUtils: attempt=0 on file=ofs://ozone1710266930/vol1/bucket1/hbase/WALs/ccycloud-2.weichiu-src.root.comops.site,22101,1
712266104607-splitting/ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266382475 after 19ms
KEY_UNDER_LEASE_SOFT_LIMIT_PERIOD org.apache.hadoop.ozone.om.exceptions.OMException: Open Key hbase/WALs/ccycloud-2.weichiu-src.root.comops.site,22101,1712266104607-splitting/ccycloud-2.weic
hiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266382475 updated recently and is inside soft limit period
        at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:756)
        at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.recoverLease(OzoneManagerProtocolClientSideTranslatorPB.java:2560)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:76)
        at com.sun.proxy.$Proxy27.recoverLease(Unknown Source)
        at org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.recoverFilePrepare(BasicRootedOzoneClientAdapterImpl.java:1384)
        at org.apache.hadoop.fs.ozone.RootedOzoneFileSystem.recoverLease(RootedOzoneFileSystem.java:154)
        at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:183)
        at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:115)
        at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:70)
        at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:455)
        at org.apache.hadoop.hbase.wal.WALSplitter.splitWAL(WALSplitter.java:306)
        at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:201)
        at org.apache.hadoop.hbase.regionserver.SplitLogWorker.splitLog(SplitLogWorker.java:108)
        at org.apache.hadoop.hbase.regionserver.SplitWALCallable.call(SplitWALCallable.java:100)
        at org.apache.hadoop.hbase.regionserver.SplitWALCallable.call(SplitWALCallable.java:46)
        at org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler.process(RSProcedureHandler.java:49)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

The Hbase code:
https://github.com/apache/hbase/blob/e48c4485db8d9255510ee1cc9cf465e14de637d7/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java#L457

note: HBase RS wal spliting opens an input stream to the WAL file. It forcefully invokes lease recovery regardless of the state of file.

Issue Links

• relates to: HDDS-10754 [hsync] lease recovery contract test class not substantiated