[IMPALA-6413] Data loading job sometimes fails

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: Impala 2.12.0
• Component/s: Infrastructure
• Epic Color: ghx-label-7
• Assignee: Philip Martin
• Reporter: Taras Bobrovytsky
• Created: 18/Jan/18 00:40
• Updated: 25/Jan/18 01:00
• Resolved: 25/Jan/18 01:00

Description

The data loading script sometimes randomly fails. I saw a failure like this on my machine and in one of our private builds.

Started Loading TPC-H data in background; pid 29360.
Loading TPC-H data (logging to /data/jenkins/workspace/impala-private-parameterized/repos/Impala/logs/data_loading/load-tpch.log)... 
Started Loading TPC-DS data in background; pid 29361.
Loading TPC-DS data (logging to /data/jenkins/workspace/impala-private-parameterized/repos/Impala/logs/data_loading/load-tpcds.log)... 
    FAILED (Took: 24 min 7 sec)
    'load-data functional-query exhaustive' failed. Tail of log:
ALTER TABLE alltypesagg ADD IF NOT EXISTS PARTITION(year=2010, month=1, day=1)

(load-functional-query-exhaustive-impala-generated-kudu-none-none.sql):
create table functional_kudu.jointbl (
  test_id bigint,
  test_name string,
  test_zip int,
  alltypes_id int,
  primary key (test_id, test_name, test_zip, alltypes_id)
)
partition by range(test_id) (partition values <= 1003, partition 1003 < values <= 1007,
partition 1007 < values) stored as kudu

(load-functional-query-exhaustive-impala-generated-seq-bzip-block.sql):
ALTER TABLE alltypesagg ADD IF NOT EXISTS PARTITION(year=2010, month=1, day=1)

(load-functional-query-exhaustive-impala-generated-seq-snap-block.sql):
ALTER TABLE alltypesagg ADD IF NOT EXISTS PARTITION(year=2010, month=1, day=1)

(load-functional-query-exhaustive-impala-generated-rc-def-block.sql):
ALTER TABLE alltypesagg ADD IF NOT EXISTS PARTITION(year=2010, month=1, day=1)

(load-functional-query-exhaustive-impala-generated-parquet-none-none.sql):
CREATE DATABASE IF NOT EXISTS functional_parquet

(load-functional-query-exhaustive-impala-generated-seq-bzip-record.sql):
USE functional_seq_record_bzip

(load-functional-query-exhaustive-impala-generated-seq-bzip-record.sql):
ALTER TABLE alltypesagg ADD IF NOT EXISTS PARTITION(year=2010, month=1, day=1)

Data Loading from Impala failed with error: ImpalaBeeswaxException:
 INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
 MESSAGE: AnalysisException: Could not resolve table reference: 'alltypesagg'

Traceback (most recent call last):
  File "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/bin/load-data.py", line 178, in exec_impala_query_from_file
    result = impala_client.execute(query)
  File "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py", line 173, in execute
    handle = self.__execute_query(query_string.strip(), user=user)
  File "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py", line 339, in __execute_query
    handle = self.execute_query_async(query_string, user=user)
  File "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py", line 335, in execute_query_async
    return self.__do_rpc(lambda: self.imp_service.query(query,))
  File "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py", line 460, in __do_rpc
    raise ImpalaBeeswaxException(self.__build_error_message(b), b)
ImpalaBeeswaxException: ImpalaBeeswaxException:
 INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
 MESSAGE: AnalysisException: Could not resolve table reference: 'alltypesagg'

Background task Loading functional-query data (pid 29359) failed.
  Loading workload 'tpch' using exploration strategy 'core' OK (Took: 25 min 26 sec)
  Loading workload 'tpcds' using exploration strategy 'core' OK (Took: 53 min 13 sec)
Error in /data/jenkins/workspace/impala-private-parameterized/repos/Impala/testdata/bin/create-load-data.sh at line 85: ;;
+ echo 'buildall.sh ' -format '-snapshot_file /data/jenkins/workspace/impala-private-parameterized/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh5-98-SNAPSHOT.tar.gz' '-metastore_snapshot_file /data/jenkins/workspace/impala-private-parameterized/testdata/hive_metastore_snapshot/hive_impala_dump_cdh5-98.txt failed.'

Issue Links

• duplicates: IMPALA-6386 Dataload can fail due to "invalidate metadata" concurrent with DDLs