[BIGTOP-3577] HBase smoke test partially fails on ppc64le

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: hbase, tests
• Assignee: Kengo Seki
• Reporter: Kengo Seki
• Created: 13/Aug/21 05:29
• Updated: 25/Aug/21 22:43
• Resolved: 25/Aug/21 21:40

Description

https://ci.bigtop.apache.org/job/Bigtop-trunk-smoke-tests/259/COMPONENTS=hdfs.hbase@hbase,OS=fedora-33-ppc64le-deploy/

The following tests fail on ppc64le for some reason.
It seems that the HFiles to be uploaded are created under /root/.gradle and not found. On other platforms, these files are created under /bigtop-home.

org.apache.bigtop.itest.hbase.smoke.TestLoadIncrementalHFiles > testSimpleLoad FAILED
    java.io.IOException: BulkLoad encountered an unrecoverable problem
        at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.bulkLoadPhase(LoadIncrementalHFiles.java:497)
        at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:428)
        at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:318)
        at org.apache.bigtop.itest.hbase.smoke.TestLoadIncrementalHFiles.runTest(TestLoadIncrementalHFiles.java:106)
        at org.apache.bigtop.itest.hbase.smoke.TestLoadIncrementalHFiles.testSimpleLoad(TestLoadIncrementalHFiles.java:52)

        Caused by:
        org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:
        Fri Aug 13 00:24:08 UTC 2021, RpcRetryingCaller{globalStartTime=1628814248730, pause=100, retries=35}, java.io.FileNotFoundException: java.io.FileNotFoundException: File file:/root/.gradle/daemon/5.6.4/testSimpleLoad_1628814207756/f1/hfile_0 does not exist
        	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:668)
        	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:989)
        	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:658)
        	at org.apache.hadoop.fs.FileSystem.access(FileSystem.java:2638)
        	at org.apache.hadoop.fs.FilterFileSystem.access(FilterFileSystem.java:469)
        	at org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:819)
        	at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:6221)
        	at org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:2428)
        	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42280)
        	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:418)
        	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
        	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
        	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)

org.apache.bigtop.itest.hbase.smoke.TestLoadIncrementalHFiles > testRegionCrossingLoad FAILED
    java.io.IOException: BulkLoad encountered an unrecoverable problem
        at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.bulkLoadPhase(LoadIncrementalHFiles.java:497)
        at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:428)
        at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:318)
        at org.apache.bigtop.itest.hbase.smoke.TestLoadIncrementalHFiles.runTest(TestLoadIncrementalHFiles.java:106)
        at org.apache.bigtop.itest.hbase.smoke.TestLoadIncrementalHFiles.testRegionCrossingLoad(TestLoadIncrementalHFiles.java:65)

        Caused by:
        org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:
        Fri Aug 13 00:33:14 UTC 2021, RpcRetryingCaller{globalStartTime=1628814794143, pause=100, retries=35}, java.io.FileNotFoundException: java.io.FileNotFoundException: File file:/root/.gradle/daemon/5.6.4/testRegionCrossingLoad_1628814759596/f1/_tmp/42c89f7f39d34657a99286ebbb8c968a.bottom does not exist
        	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:668)
        	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:989)
        	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:658)
        	at org.apache.hadoop.fs.FileSystem.access(FileSystem.java:2638)
        	at org.apache.hadoop.fs.FilterFileSystem.access(FilterFileSystem.java:469)
        	at org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:819)
        	at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:6221)
        	at org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:2428)
        	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42280)
        	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:418)
        	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
        	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
        	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)

Issue Links

• links to: https://github.com/apache/bigtop/pull/814