[HDDS-8561] [Hbase-Ozone] RS Aborting after "Open file cannot be renamed" Error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.4.0
• Component/s: Ozone Manager
• Assignee: Unassigned
• Reporter: Pratyush Bhatt
• Created: 07/May/23 06:22
• Updated: 12/Apr/24 18:06
• Resolved: 31/May/23 14:00

Description

Region server Aborted after WAL renaming failed with:
RENAME_OPEN_FILE org.apache.hadoop.ozone.om.exceptions.OMException: Open file cannot be renamed
hbase.regionserver.walroll.archive.retries was set to 10.

2023-05-05 22:16:42,080 ERROR org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL: Failed log archiving for the log ofs://ozone1/vol1/bucket1/hbase4/WALs/ozone-new22-1.ozone-new22.root.hwx.site,22101,1683321374304/ozone-new22-1.ozone-new22.root.hwx.site%2C22101%2C1683321374304.ozone-new22-1.ozone-new22.root.hwx.site%2C22101%2C1683321374304.regiongroup-0.1683321395004,
RENAME_OPEN_FILE org.apache.hadoop.ozone.om.exceptions.OMException: Open file cannot be renamed
        at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:711)
        at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.renameKey(OzoneManagerProtocolClientSideTranslatorPB.java:882)
        at org.apache.hadoop.ozone.client.rpc.RpcClient.renameKey(RpcClient.java:1503)
        at org.apache.hadoop.ozone.client.OzoneBucket.renameKey(OzoneBucket.java:611)
        at org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.rename(BasicRootedOzoneClientAdapterImpl.java:481)
        at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.renameFSO(BasicRootedOzoneFileSystem.java:446)
        at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.rename(BasicRootedOzoneFileSystem.java:359)
        at org.apache.hadoop.hbase.util.CommonFSUtils.renameAndSetModifyTime(CommonFSUtils.java:711)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.archiveLogFile(AbstractFSWAL.java:741)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.archive(AbstractFSWAL.java:705)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$cleanOldLogs$1(AbstractFSWAL.java:694)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2023-05-05 22:16:42,085 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ***** ABORTING region server ozone-new22-1.ozone-new22.root.hwx.site,22101,1683321374304: Failed log archiving *****
RENAME_OPEN_FILE org.apache.hadoop.ozone.om.exceptions.OMException: Open file cannot be renamed
        at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:711)
        at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.renameKey(OzoneManagerProtocolClientSideTranslatorPB.java:882)
        at org.apache.hadoop.ozone.client.rpc.RpcClient.renameKey(RpcClient.java:1503)
        at org.apache.hadoop.ozone.client.OzoneBucket.renameKey(OzoneBucket.java:611)
        at org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.rename(BasicRootedOzoneClientAdapterImpl.java:481)
        at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.renameFSO(BasicRootedOzoneFileSystem.java:446)
        at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.rename(BasicRootedOzoneFileSystem.java:359)
        at org.apache.hadoop.hbase.util.CommonFSUtils.renameAndSetModifyTime(CommonFSUtils.java:711)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.archiveLogFile(AbstractFSWAL.java:741)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.archive(AbstractFSWAL.java:705)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$cleanOldLogs$1(AbstractFSWAL.java:694)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2023-05-05 22:16:42,085 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor, org.apache.hadoop.hbase.security.token.TokenProvider, org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint]

Issue Links

• is caused by: HDDS-8545 [hsync] reject renaming open file
• is fixed by: HBASE-27732 NPE in TestBasicWALEntryStreamFSHLog.testEOFExceptionInOldWALsDirectory