[IMPALA-3802] test_create_alter_bulk_partition failure on Isilon: EOFException during ALTER TABLE

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 2.6.0
• Fix Version/s: Impala 2.8.0
• Component/s: Catalog
• Labels: test_issue
• Target Version: Impala 2.8.0
• Assignee: Taras Bobrovytsky
• Reporter: Michael Brown
• Created: 28/Jun/16 23:02
• Updated: 08/Dec/16 08:23
• Resolved: 08/Dec/16 08:23

Description

tarasbob can you give a first look over this? Given the flakiness of the Isilon cluster, it's hard to know whether this is a product issue, test issue, or environment issue.

http://sandbox.jenkins.cloudera.com/job/impala-umbrella-build-and-test-isilon/109/

 TestDdlStatements.test_create_alter_bulk_partition[exec_option: {'batch_size': 0, 'num_nodes': 0, 'sync_ddl': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none] 
metadata/test_ddl.py:496: in test_create_alter_bulk_partition
    i))
common/impala_connection.py:161: in execute
    return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:163: in execute
    handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:327: in __execute_query
    handle = self.execute_query_async(query_string, user=user)
beeswax/impala_beeswax.py:323: in execute_query_async
    return self.__do_rpc(lambda: self.imp_service.query(query,))
beeswax/impala_beeswax.py:448: in __do_rpc
    raise ImpalaBeeswaxException(self.__build_error_message(b), b)
E   ImpalaBeeswaxException: ImpalaBeeswaxException:
E    INNER EXCEPTION: <class 'impala._thrift_gen.beeswax.ttypes.BeeswaxException'>
E    MESSAGE: 
E   CatalogException: Failed to retrieve file descriptors from path hdfs://10.17.95.15:8020/test-warehouse/foo_part/j=1/s=1
E   CAUSED BY: EOFException: End of File Exception between local host is: "impala-boost-static-burst-slave-0a66.vpc.cloudera.com/127.0.0.1"; destination host is: "10.17.95.15":8020; : java.io.EOFException; For more details see:  http://wiki.apache.org/hadoop/EOFException
E   CAUSED BY: EOFException: null

The exception in that stack is the same one printed in the catalog logs.

-- executing against localhost:21000
use default;

-- executing against localhost:21000
drop table if exists foo_part;

-- executing against localhost:21000
create table foo_part(i int) partitioned by(j int, s string)
         location '/test-warehouse/foo_part';

-- executing against localhost:21000
alter table foo_part add partition(j=0, s='0');

MainThread: ADD PARTITION #0 exec time: 10.0077879429
-- executing against localhost:21000
alter table foo_part add partition(j=1, s='1');

This last alter table was the failure.