[HBASE-27061] two phase bulkload is broken when SFT is in use.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.5.0, 3.0.0-alpha-2, 2.6.0
• Fix Version/s: 2.5.0, 3.0.0-alpha-3
• Assignee: Sergey Soldatov
• Reporter: Sergey Soldatov
• Created: 25/May/22 04:13
• Updated: 01/Sep/22 10:57
• Resolved: 26/May/22 13:12

Description

In HBASE-26707 for the SFT case, we are writing files directly to the region location. For that we are using HRegion.regionDir as the staging directory. The problem is that in reality, this dir is pointing to the WAL dir, so for S3 deployments that would be pointing to the hdfs. As the result during the execution of LoadIncrementalHFiles the process failed with the exception:

2022-05-24 03:31:23,656 ERROR org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager: Failed to complete bulk load
java.lang.IllegalArgumentException: Wrong FS hdfs://ns1//hbase-wals/data/default/employees/4f367b303da4fed7667fff07fd4c6066/department/acd971097924463da6d6e3a15f9527da -expected s3a://hbase
        at org.apache.hadoop.fs.s3native.S3xLoginHelper.checkPath(S3xLoginHelper.java:224)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.checkPath(S3AFileSystem.java:1375)
        at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:647)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.makeQualified(S3AFileSystem.java:1337)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.qualify(S3AFileSystem.java:1363)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3521)
        at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:511)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:397)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:387)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
        at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$SecureBulkLoadListener.prepareBulkLoad(SecureBulkLoadManager.java:397)
        at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:6994)
        at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:291)
        at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:266)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1879)
        at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager.secureBulkLoadHFiles(SecureBulkLoadManager.java:266)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:2453)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45821)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:392)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:140)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:359)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:339)

 

Issue Links

• links to: GitHub Pull Request #4462
• links to: GitHub Pull Request #4465