[IMPALA-8120] Build Failure: Undetermined data load error for tpcds for Kudu

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: Impala 3.1.0
• Fix Version/s: Impala 3.1.0
• Component/s: Infrastructure
• Labels: build-failure
• Epic Color: ghx-label-7
• Assignee: Paul Rogers
• Reporter: Paul Rogers
• Created: 25/Jan/19 19:12
• Updated: 29/Jan/19 00:53
• Resolved: 29/Jan/19 00:53

Description

The build for the latest master failed with an unspecified error when loading data:

03:07:58 Loading TPC-DS data (logging to /data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/logs/data_loading/load-tpcds.log)... 
03:07:58 + echo 'Log for command '\''load-data' tpcds 'core'\'''
03:07:58 + START_TIME=38
03:07:58 + load-data tpcds core
03:11:31 + ELAPSED_TIME=213
03:11:31 + echo '    FAILED (Took: 3 min 33 sec)'
03:11:31     FAILED (Took: 3 min 33 sec)

Also:

03:17:43 + echo '  Loading workload '\''tpcds'\'' using exploration strategy '\''core'\'' OK (Took: 9 min 45 sec)'
03:17:43   Loading workload 'tpcds' using exploration strategy 'core' OK (Took: 9 min 45 sec)
03:19:39 + ELAPSED_TIME=701
03:19:39 + echo '    FAILED (Took: 11 min 41 sec)'
03:19:39     FAILED (Took: 11 min 41 sec)
03:19:39 + echo '    '\''load-data' functional-query 'exhaustive'\'' failed. Tail of log:'

Tail of log:

03:19:39 03:19:39 Error executing impala SQL: /data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql See: /data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log
...
03:19:39 ++ report_build_error 85
03:19:39 +++ cd /data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala
03:19:39 +++ awk 'NR == 85' /data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/testdata/bin/create-load-data.sh
03:19:39 ++ ERROR_MSG='    -cm_host)'
03:19:39 +++ basename -- /data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/testdata/bin/create-load-data.sh
03:19:39 ++ FILENAME=create-load-data.sh
03:19:39 ++ echo ERROR in /data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/testdata/bin/create-load-data.sh at line 85: '-cm_host)'
...
03:19:39 + [[ RELEASE = \A\S\A\N ]]

From the catalogd log:

I0125 03:11:31.808512  9906 jni-util.cc:256] org.apache.impala.common.ImpalaRuntimeException: Error creating Kudu table 'impala::tpch_kudu.lineitem'
        at org.apache.impala.service.KuduCatalogOpExecutor.createManagedTable(KuduCatalogOpExecutor.java:94)
        at org.apache.impala.service.CatalogOpExecutor.createKuduTable(CatalogOpExecutor.java:1768)
        at org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:1686)
        at org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:287)
        at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:154)
Caused by: org.apache.kudu.client.NonRecoverableException: can not complete before timeout: KuduRpc(method=ListTables, tablet=null, attempt=95, DeadlineTracker(timeout=180000, elapsed=178163), Traces: [0ms] querying master, [41ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [91ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [98ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [118ms] querying master, [121ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [122ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [122ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [138ms] querying master, [138ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [139ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [140ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [159ms] querying master, [159ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [160ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [161ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [178ms] querying master, [178ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [179ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [180ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [219ms] querying master, [219ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [220ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [220ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [258ms] querying master, [258ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [259ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [260ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [319ms] querying master, [319ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [320ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [320ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [358ms] querying master, [358ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [359ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [360ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [879ms] querying master, [879ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [880ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [881ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [1278ms] querying master, [1278ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [1279ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [1280ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [2139ms] querying master, [2139ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [2140ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [2141ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [5058ms] querying master, [5059ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [5060ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [5060ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [8239ms] querying master, [8239ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [8240ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [8241ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [11539ms] querying master, [11539ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [11540ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [11540ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [13258ms] querying master, [13259ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [13259ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [13260ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [14239ms] querying master, [14239ms] Sub rpc: ConnectToMaster sending RPC to server master-localhost:7051, [14240ms] Sub rpc: ConnectToMaster received from server master-localhost:7051 response Network error: java.net.ConnectException: Connection refused: localhost/127.0.0.1:7051, [14241ms] delaying RPC due to Service unavailable: Master config (localhost:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: loc

Issue Links

• duplicates: IMPALA-8091 minicluster kudu failure: Cannot initialize clock: failed to wait for clock sync