[HBASE-22200] WALSplitter.hasRecoveredEdits should use same FS instance from WAL region dir

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.2.0, 2.0.6, 2.1.5
• Component/s: wal
• Labels: S3 WAL
• Hadoop Flags: Reviewed
• Assignee: Wellington Chevreuil
• Reporter: Wellington Chevreuil
• Created: 10/Apr/19 12:19
• Updated: 24/Jun/22 17:52
• Resolved: 22/Apr/19 23:22

Description

WALSplitter.hasRecoveredEdits should use same FS instance from WAL region dir when checking for recovered.edits files, instead of taking FS instance as additional method parameter. When specifying different file systems for wal dir and root dir, WALSplitter.hasRecoveredEdits current implementation will crash or give wrong results. As of now, it's being used indirectly by SplitTableRegionProcedure. When running tests with WAL dir on HDFS and root dir on S3, for example, noticed region split failing with below error:

2019-04-08 13:53:58,064 ERROR org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught runtime exception: pid=98, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=test-tbl, parent=4c5db01611e97e3abbe02e781e867212, daughterA=28a0a5e4ef7618899f6bd6dfb5335fe7, daughterB=05fa26feaf03ebf9e87e099cbd1eabac
java.lang.IllegalArgumentException: Path hdfs://host-1.example.com:8020/wal_dir/default/test-tbl/4c5db01611e97e3abbe02e781e867212/recovered.edits scheme must be s3a
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
        at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.checkPath(DynamoDBMetadataStore.java:1127)
        at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.get(DynamoDBMetadataStore.java:437)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2110)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668)
        at org.apache.hadoop.hbase.wal.WALSplitter.getSplitEditFilesSorted(WALSplitter.java:576)
        at org.apache.hadoop.hbase.wal.WALSplitter.hasRecoveredEdits(WALSplitter.java:558)
        at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.hasRecoveredEdits(SplitTableRegionProcedure.java:148)
        at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:255)

Since WALSplitter.hasRecoveredEdits already resolves the proper WAL dir for the region, we can simply re-use FS instance from the path instance for the WAL dir region, when searching for recovered.edits.

Issue Links

• is related to: HBASE-20734 Colocate recovered edits directory with hbase.wal.dir
• relates to: HBASE-21688 Address WAL filesystem issues