[IMPALA-6580] Cannot load Kudu functional data on localfs build

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 3.0, Impala 2.12.0
• Fix Version/s: Impala 2.12.0
• Component/s: Infrastructure
• Labels: broken-build
• Target Version: Impala 3.0, Impala 2.12.0
• Epic Color: ghx-label-1
• Assignee: Joe McDonnell
• Reporter: Tim Armstrong
• Created: 23/Feb/18 23:13
• Updated: 25/Feb/18 17:25
• Resolved: 25/Feb/18 17:25

Description

Data loading fails here:

.
08:15:58 Skipping table: functional_kudu.kite_required_fields, 'only' constraint for format did not include this table.
08:15:58 Skipping table: functional_kudu.bad_column_metadata, 'only' constraint for format did not include this table.
08:15:58 Skipping table: functional_kudu.bad_serde, 'only' constraint for format did not include this table.
08:15:58 Skipping table: functional_kudu.rcfile_lazy_binary_serde, 'only' consmkdir: `/test-warehouse/decimal_tbl/d6=1': Input/output error
08:15:58 traint for format did not include this table.
08:15:58 Traceback (most recent call last):
08:15:58   File "/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/testdata/bin/generate-schema-statements.py", line 763, in <module>
08:15:58     test_vectors, sections, include_constraints, exclude_constraints, only_constraints)
08:15:58   File "/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/testdata/bin/generate-schema-statements.py", line 534, in generate_statements
08:15:58     load = eval_section(section['LOAD'])
08:15:58   File "/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/testdata/bin/generate-schema-statements.py", line 473, in eval_section
08:15:58     assert p.returncode == 0
08:15:58 AssertionError
08:15:58 Error generating schema statements for workload: functional-query
08:15:58 Background task Loading Kudu functional (pid 26145) failed.

The problem is that the "LOAD" section is being executed for Kudu

====
---- DATASET
functional
---- BASE_TABLE_NAME
decimal_tbl
---- COLUMNS
d1 DECIMAL
d2 DECIMAL(10, 0)
d3 DECIMAL(20, 10)
d4 DECIMAL(38, 38)
d5 DECIMAL(10, 5)
---- PARTITION_COLUMNS
d6 DECIMAL(9, 0)
---- ALTER
ALTER TABLE {table_name} ADD IF NOT EXISTS PARTITION(d6=1);
---- ROW_FORMAT
delimited fields terminated by ','
---- LOAD
`hadoop fs -mkdir -p /test-warehouse/decimal_tbl/d6=1 && hadoop fs -put -f \
${IMPALA_HOME}/testdata/data/decimal_tbl.txt /test-warehouse/decimal_tbl/d6=1/
---- DEPENDENT_LOAD
INSERT OVERWRITE TABLE {db_name}{db_suffix}.{table_name} partition(d6)
select * from functional.{table_name};
---- CREATE_KUDU
DROP TABLE IF EXISTS {db_name}{db_suffix}.{table_name};
CREATE TABLE {db_name}{db_suffix}.{table_name} (
  d1 DECIMAL,
  d2 DECIMAL(10, 0),
  d3 DECIMAL(20, 10),
  d4 DECIMAL(38, 38),
  d5 DECIMAL(10, 5),
  d6 DECIMAL(9, 0),
  PRIMARY KEY (d1, d2, d3, d4, d5, d6)
)
PARTITION BY HASH PARTITIONS 3
STORED AS KUDU;
---- DEPENDENT_LOAD_KUDU
INSERT into TABLE {db_name}{db_suffix}.{table_name}
SELECT d1, d2, d3, d4, d5, d6
FROM {db_name}.{table_name};

Issue Links

• is blocked by: IMPALA-5752 Add support for DECIMAL on Kudu tables