[HBASE-23197] "IllegalArgumentException: Wrong FS" on edits replay when WALs on different file system and hbase.region.archive.recovered.edits is enabled.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1
• Fix Version/s: 3.0.0-alpha-1, 2.3.0, 2.2.3, 2.1.9
• Assignee: Wellington Chevreuil
• Reporter: Wellington Chevreuil
• Created: 21/Oct/19 15:18
• Updated: 26/Nov/19 23:42
• Resolved: 26/Nov/19 15:49

Description

This was caught by execution tests from our romil.choksi, when running hbase with WALs on hdfs, HFiles on S3. Upon initial analysis from elserj of below stack trace, we concluded this is triggered by enabling hbase.region.archive.recovered.edits property.

***** ABORTING region server my-rs-1.example.com,16020,1571335836583: Failed to open region hbase:meta,,1.1588230740 and can not recover *****
Cause:
java.lang.IllegalArgumentException: Wrong FS: s3a://qe-s3-bucket-mst/cc-hbase-191017-113225-e6x-dw-4/hbase/archive/data/hbase/meta/1588230740/recovered.edits, expected: hdfs://my-hdfs-1.example.com:8020
        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:742)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:234)
        at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1323)
        at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1320)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1337)
        at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1312)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2292)
        at org.apache.hadoop.hbase.backup.HFileArchiver.archiveStoreFiles(HFileArchiver.java:321)
        at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.removeStoreFiles(HRegionFileSystem.java:505)
        at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:4626)
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:968)
        at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:915)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7320) 

This property is available mainly for debugging purposes, and it wraps recovered edits over store files for re-using the StoreFiles archiving logic.

Issue Links

• links to: GitHub Pull Request #740