[IMPALA-6776] Failed to assign hbase regions to servers

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 3.0
• Fix Version/s: Impala 2.13.0, Impala 3.1.0
• Component/s: Infrastructure
• Labels: broken-build
• Epic Color: ghx-label-7
• Assignee: Vuk Ercegovac
• Reporter: Tianyi Wang
• Created: 30/Mar/18 18:52
• Updated: 29/May/18 20:32
• Resolved: 29/May/18 20:32

Description

After switching to hadoop 3 components, split-hbase.sh failed in HBaseTestDataRegionAssigment:

20:40:27 Splitting HBase (logging to /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/create-hbase.log)... 
20:41:51     FAILED (Took: 1 min 24 sec)
20:41:51     '/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/testdata/bin/split-hbase.sh' failed. Tail of log:
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,3,1522381286429.7b13fefeda7afac230e22150deab5266. 3 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,5,1522381287511.7a243a822c5c4844a2a3d0f67a541961. 5 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,7,1522381288718.80d6e4a799ad114a146dc3cb41e18e93. 7 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,9,1522381288718.d705a2ea635916f4bb510ca60764080a. 9 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,,1522381282868.a99b569f5417ea9e2561eb5566c31be0.  -> localhost:16203, expecting localhost,16201,1522374371810
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,1,1522381285023.5fb566ba94e5fbb8aeca39f3da0a6362. 1 -> localhost:16201, expecting localhost,16201,1522374371810
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,3,1522381286429.7b13fefeda7afac230e22150deab5266. 3 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,5,1522381287511.7a243a822c5c4844a2a3d0f67a541961. 5 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,7,1522381288718.80d6e4a799ad114a146dc3cb41e18e93. 7 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,9,1522381288718.d705a2ea635916f4bb510ca60764080a. 9 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,,1522381282868.a99b569f5417ea9e2561eb5566c31be0.  -> localhost:16203, expecting localhost,16201,1522374371810
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,1,1522381285023.5fb566ba94e5fbb8aeca39f3da0a6362. 1 -> localhost:16201, expecting localhost,16201,1522374371810
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,3,1522381286429.7b13fefeda7afac230e22150deab5266. 3 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,5,1522381287511.7a243a822c5c4844a2a3d0f67a541961. 5 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,7,1522381288718.80d6e4a799ad114a146dc3cb41e18e93. 7 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 18/03/29 20:41:50 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,9,1522381288718.d705a2ea635916f4bb510ca60764080a. 9 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,,1522381282868.a99b569f5417ea9e2561eb5566c31be0.  -> localhost:16203, expecting localhost,16201,1522374371810
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,1,1522381285023.5fb566ba94e5fbb8aeca39f3da0a6362. 1 -> localhost:16201, expecting localhost,16201,1522374371810
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,3,1522381286429.7b13fefeda7afac230e22150deab5266. 3 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,5,1522381287511.7a243a822c5c4844a2a3d0f67a541961. 5 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,7,1522381288718.80d6e4a799ad114a146dc3cb41e18e93. 7 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,9,1522381288718.d705a2ea635916f4bb510ca60764080a. 9 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,,1522381282868.a99b569f5417ea9e2561eb5566c31be0.  -> localhost:16203, expecting localhost,16201,1522374371810
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,1,1522381285023.5fb566ba94e5fbb8aeca39f3da0a6362. 1 -> localhost:16201, expecting localhost,16201,1522374371810
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,3,1522381286429.7b13fefeda7afac230e22150deab5266. 3 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,5,1522381287511.7a243a822c5c4844a2a3d0f67a541961. 5 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,7,1522381288718.80d6e4a799ad114a146dc3cb41e18e93. 7 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,9,1522381288718.d705a2ea635916f4bb510ca60764080a. 9 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,,1522381282868.a99b569f5417ea9e2561eb5566c31be0.  -> localhost:16203, expecting localhost,16201,1522374371810
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,1,1522381285023.5fb566ba94e5fbb8aeca39f3da0a6362. 1 -> localhost:16201, expecting localhost,16201,1522374371810
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,3,1522381286429.7b13fefeda7afac230e22150deab5266. 3 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,5,1522381287511.7a243a822c5c4844a2a3d0f67a541961. 5 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,7,1522381288718.80d6e4a799ad114a146dc3cb41e18e93. 7 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,9,1522381288718.d705a2ea635916f4bb510ca60764080a. 9 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,,1522381282868.a99b569f5417ea9e2561eb5566c31be0.  -> localhost:16203, expecting localhost,16201,1522374371810
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,1,1522381285023.5fb566ba94e5fbb8aeca39f3da0a6362. 1 -> localhost:16201, expecting localhost,16201,1522374371810
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,3,1522381286429.7b13fefeda7afac230e22150deab5266. 3 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,5,1522381287511.7a243a822c5c4844a2a3d0f67a541961. 5 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,7,1522381288718.80d6e4a799ad114a146dc3cb41e18e93. 7 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,9,1522381288718.d705a2ea635916f4bb510ca60764080a. 9 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,,1522381282868.a99b569f5417ea9e2561eb5566c31be0.  -> localhost:16203, expecting localhost,16201,1522374371810
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,1,1522381285023.5fb566ba94e5fbb8aeca39f3da0a6362. 1 -> localhost:16201, expecting localhost,16201,1522374371810
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,3,1522381286429.7b13fefeda7afac230e22150deab5266. 3 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,5,1522381287511.7a243a822c5c4844a2a3d0f67a541961. 5 -> localhost:16202, expecting localhost,16202,1522374373018
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,7,1522381288718.80d6e4a799ad114a146dc3cb41e18e93. 7 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 18/03/29 20:41:51 INFO datagenerator.HBaseTestDataRegionAssigment: functional_hbase.alltypessmall,9,1522381288718.d705a2ea635916f4bb510ca60764080a. 9 -> localhost:16203, expecting localhost,16203,1522374374705
20:41:51 Exception in thread "main" java.lang.IllegalStateException: Failed to assign regions to servers after 10 seconds.
20:41:51 	at org.apache.impala.datagenerator.HBaseTestDataRegionAssigment.performAssigment(HBaseTestDataRegionAssigment.java:192)
20:41:51 	at org.apache.impala.datagenerator.HBaseTestDataRegionAssigment.main(HBaseTestDataRegionAssigment.java:323)
20:41:51 Error in /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/testdata/bin/split-hbase.sh at line 41: "$JAVA" ${JAVA_KERBEROS_MAGIC} \
20:41:51 Error in /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/testdata/bin/create-load-data.sh at line 48: LOAD_DATA_ARGS=""
20:41:51 + echo 'buildall.sh ' -release -format '-testdata failed.'
20:41:51 buildall.sh  -release -format -testdata failed.

Issue Links

• is duplicated by: IMPALA-6938 Build failing because failed to assign HBase regions during data load
• is related to: IMPALA-7061 PlannerTests should assign HBase splits as part of the test
• relates to: IMPALA-7043 Failure in HBase splitting should not fail dataload