[HBASE-17429] HBase bulkload cannot support HDFS viewFs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.98.6.1, 1.2.4
• Fix Version/s: 1.4.0, 2.0.0

Environment

CDH5.7.0 hbase0.98.6

• Hadoop Flags: Reviewed
• Assignee: shenxianqiang
• Reporter: shenxianqiang
• Created: 05/Jan/17 08:21
• Updated: 10/Jan/17 06:08
• Resolved: 10/Jan/17 02:54

Description

Since hadoop cluster suppor fedration,hbase bulkload performance degrades dramatically. Even if hbase directory and bulkload directory in the same nameservice.

2017-01-04 21:58:40,919 INFO org.apache.hadoop.hbase.regionserver.HRegionFileSystem: use bulkload file name is : hdfs://CloudTestNameNode2:8020
2017-01-04 21:58:40,924 ERROR org.apache.hadoop.ipc.RpcServer: Unexpected throwable object
java.lang.IllegalArgumentException: Wrong FS: viewfs://nsX/user/test/I/_tmp/9cde5dde60374b1483b7d09b65258304.top, expected: hdfs://CloudTestNameNode2:8020
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:425)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:373)
at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:496)
at org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:708)
at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3658)
at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3564)
at org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegionServer.java:3378)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29589)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:745)