[IMPALA-5608] test_huge_num_rows failure in hdfs dfs -copyFromLocal on S3

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 2.10.0
• Component/s: Infrastructure
• Labels: flaky
• Epic Color: ghx-label-1
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 30/Jun/17 17:17
• Updated: 15/Sep/17 21:20
• Resolved: 15/Sep/17 21:20

Description

This may just be a temporary infra problem but filing a JIRA so we can figure out if it repeats.

09:13:07 FAIL query_test/test_scanners.py::TestParquet::()::test_huge_num_rows[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
09:13:07 =================================== FAILURES ===================================
09:13:07  TestParquet.test_huge_num_rows[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
09:13:07 [gw2] linux2 -- Python 2.6.6 /data/jenkins/workspace/impala-umbrella-build-and-test-s3/repos/Impala/bin/../infra/python/env/bin/python
09:13:07 query_test/test_scanners.py:312: in test_huge_num_rows
09:13:07     huge_num_rows_loc])
09:13:07 /usr/lib64/python2.6/subprocess.py:505: in check_call
09:13:07     raise CalledProcessError(retcode, cmd)
09:13:07 E   CalledProcessError: Command '['hdfs', 'dfs', '-copyFromLocal', '/path/to/Impala/testdata/data/huge_num_rows.parquet', 's3a://bucket-name/test-warehouse/test_huge_num_rows_5c0c9da.db/huge_num_rows']' returned non-zero exit status 1
09:13:07 ---------------------------- Captured stderr setup -----------------------------
09:13:07 SET sync_ddl=False;
09:13:07 -- executing against localhost:21000
09:13:07 DROP DATABASE IF EXISTS `test_huge_num_rows_5c0c9da` CASCADE;
09:13:07 
09:13:07 SET sync_ddl=False;
09:13:07 -- executing against localhost:21000
09:13:07 CREATE DATABASE `test_huge_num_rows_5c0c9da`;
09:13:07 
09:13:07 MainThread: Created database "test_huge_num_rows_5c0c9da" for test ID "query_test/test_scanners.py::TestParquet::()::test_huge_num_rows[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]"
09:13:07 ----------------------------- Captured stderr call -----------------------------
09:13:07 -- executing against localhost:21000
09:13:07 create table test_huge_num_rows_5c0c9da.huge_num_rows (i int) stored as parquet;
09:13:07 
09:13:07 copyFromLocal: rename `s3a://bucket-name/test-warehouse/test_huge_num_rows_5c0c9da.db/huge_num_rows/huge_num_rows.parquet._COPYING_' to `s3a://bucket-name/test-warehouse/test_huge_num_rows_5c0c9da.db/huge_num_rows/huge_num_rows.parquet': Input/output error