Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4567

TestKuduOperations.test_kudu_alter_table fails

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Backend
    • Labels:

      Description

      The exhaustive builds on RHEL7 hits the following error.

      Stacktrace
      
      query_test/test_kudu.py:62: in test_kudu_alter_table
          self.run_test_case('QueryTest/kudu_alter', vector, use_db=unique_database)
      common/impala_test_suite.py:321: in run_test_case
          result = self.__execute_query(target_impalad_client, query, user=user)
      common/impala_test_suite.py:525: in __execute_query
          return impalad_client.execute(query, user=user)
      common/impala_connection.py:160: in execute
          return self.__beeswax_client.execute(sql_stmt, user=user)
      beeswax/impala_beeswax.py:173: in execute
          handle = self.__execute_query(query_string.strip(), user=user)
      beeswax/impala_beeswax.py:337: in __execute_query
          handle = self.execute_query_async(query_string, user=user)
      beeswax/impala_beeswax.py:333: in execute_query_async
          return self.__do_rpc(lambda: self.imp_service.query(query,))
      beeswax/impala_beeswax.py:458: in __do_rpc
          raise ImpalaBeeswaxException(self.__build_error_message(b), b)
      E   ImpalaBeeswaxException: ImpalaBeeswaxException:
      E    INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
      E    MESSAGE: AnalysisException: Target table 'test_kudu_alter_table_8df0adc5.tbl_to_alter' has fewer columns (3) than the SELECT / VALUES clause returns (5)
      Standard Error
      
      SET sync_ddl=False;
      -- executing against localhost:21000
      DROP DATABASE IF EXISTS `test_kudu_alter_table_8df0adc5` CASCADE;
      
      SET sync_ddl=False;
      -- executing against localhost:21000
      CREATE DATABASE `test_kudu_alter_table_8df0adc5`;
      
      MainThread: Created database "test_kudu_alter_table_8df0adc5" for test ID "query_test/test_kudu.py::TestKuduOperations::()::test_kudu_alter_table[exec_option: {'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none]"
      -- executing against localhost:21000
      use test_kudu_alter_table_8df0adc5;
      
      SET disable_codegen=True;
      SET abort_on_error=1;
      SET exec_single_node_rows_threshold=0;
      SET batch_size=0;
      SET num_nodes=0;
      -- executing against localhost:21000
      create table simple (id int primary key, name string, valf float, vali bigint)
        distribute by hash (id) into 3 buckets stored as kudu;
      
      -- executing against localhost:21000
      alter table simple set tblproperties (
        'kudu.master_addresses' = 'localhost'
      );
      
      -- executing against localhost:21000
      describe formatted simple;
      
      -- executing against localhost:21000
      alter table simple set tblproperties ('kudu.master_addresses' = '127.0.0.1');
      
      -- executing against localhost:21000
      alter table simple set tblproperties ('kudu.master_addresses' = 'invalid_host');
      
      -- executing against localhost:21000
      alter table simple rename to simple_new;
      
      -- executing against localhost:21000
      select count(*) from simple_new;
      
      -- executing against localhost:21000
      create table tbl_to_alter (id int primary key, name string null, vali bigint not null)
        distribute by range (id) (partition 1 < values <= 10) stored as kudu
        tblproperties('kudu.table_name'='tbl_to_alter');
      
      -- executing against localhost:21000
      alter table tbl_to_alter add range partition 10 < values <= 20;
      
      -- executing against localhost:21000
      insert into tbl_to_alter values (15, 'name', 100);
      
      -- executing against localhost:21000
      select * from tbl_to_alter limit 1000;
      
      -- executing against localhost:21000
      alter table tbl_to_alter add range partition value = 100;
      
      -- executing against localhost:21000
      insert into tbl_to_alter values (100, 'name1', 1000);
      
      -- executing against localhost:21000
      select * from tbl_to_alter limit 1000;
      
      -- executing against localhost:21000
      alter table tbl_to_alter add range partition 1000 < values;
      
      -- executing against localhost:21000
      alter table tbl_to_alter add range partition 10 < values <= 30;
      
      -- executing against localhost:21000
      alter table tbl_to_alter add if not exists range partition 10 < values <= 30;
      
      -- executing against localhost:21000
      alter table tbl_to_alter drop range partition value = 100;
      
      -- executing against localhost:21000
      select * from tbl_to_alter;
      
      -- executing against localhost:21000
      alter table tbl_to_alter drop range partition 10 < values <= 20;
      
      -- executing against localhost:21000
      alter table tbl_to_alter drop range partition 1 < values <= 10;
      
      -- executing against localhost:21000
      
      alter table tbl_to_alter drop range partition 1000 < values;
      
      -- executing against localhost:21000
      select count(*), count(id) from tbl_to_alter
        where id = 1 and cast(sin(id) as boolean) = true;
      
      -- executing against localhost:21000
      insert into tbl_to_alter values (1, 'name', 100);
      
      -- executing against localhost:21000
      alter table tbl_to_alter add range partition 1 < values <= 20;
      
      -- executing against localhost:21000
      
      alter table tbl_to_alter add columns (new_col1 int not null default 10,
        new_col2 bigint not null default 1000);
      
      -- executing against localhost:21000
      insert into tbl_to_alter values (2, 'test', 100, 1, 100);
      

        Activity

        Hide
        kwho Michael Ho added a comment -

        The build is at commit d21728c0ae4ddcd1a063341d5f2db98534aedf3a

        Show
        kwho Michael Ho added a comment - The build is at commit d21728c0ae4ddcd1a063341d5f2db98534aedf3a
        Hide
        kwho Michael Ho added a comment -

        Another failure:

        11:39:35 =================================== FAILURES ===================================
        11:39:35  TestKuduOperations.test_kudu_alter_table[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] 
        11:39:35 [gw3] linux2 -- Python 2.6.6 /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/bin/../infra/python/env/bin/python
        11:39:35 query_test/test_kudu.py:62: in test_kudu_alter_table
        11:39:35     self.run_test_case('QueryTest/kudu_alter', vector, use_db=unique_database)
        11:39:35 common/impala_test_suite.py:368: in run_test_case
        11:39:35     pytest.config.option.update_results, result_section='DML_RESULTS')
        11:39:35 common/test_result_verifier.py:404: in verify_raw_results
        11:39:35     VERIFIER_MAP[verifier](expected, actual)
        11:39:35 common/test_result_verifier.py:231: in verify_query_result_is_equal
        11:39:35     assert expected_results == actual_results
        11:39:35 E   assert Comparing QueryTestResults (expected vs actual):
        11:39:35 E     15,'name',100 != None
        11:39:35 E     Number of rows returned (expected vs actual): 1 != 0
        11:39:35 ---------------------------- Captured stderr setup -----------------------------
        11:39:35 SET sync_ddl=False;
        11:39:35 -- executing against localhost:21000
        11:39:35 DROP DATABASE IF EXISTS `test_kudu_alter_table_ddaaa05e` CASCADE;
        11:39:35 
        11:39:35 SET sync_ddl=False;
        11:39:35 -- executing against localhost:21000
        11:39:35 CREATE DATABASE `test_kudu_alter_table_ddaaa05e`;
        11:39:35 
        11:39:35 MainThread: Created database "test_kudu_alter_table_ddaaa05e" for test ID "query_test/test_kudu.py::TestKuduOperations::()::test_kudu_alter_table[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none]"
        11:39:35 ----------------------------- Captured stderr call -----------------------------
        11:39:35 -- executing against localhost:21000
        11:39:35 use test_kudu_alter_table_ddaaa05e;
        11:39:35 
        11:39:35 SET disable_codegen=False;
        11:39:35 SET abort_on_error=1;
        11:39:35 SET exec_single_node_rows_threshold=0;
        11:39:35 SET batch_size=0;
        11:39:35 SET num_nodes=0;
        11:39:35 -- executing against localhost:21000
        11:39:35 create table simple (id int primary key, name string, valf float, vali bigint)
        11:39:35   distribute by hash (id) into 3 buckets stored as kudu;
        11:39:35 
        11:39:35 -- executing against localhost:21000
        11:39:35 alter table simple set tblproperties (
        11:39:35   'kudu.master_addresses' = 'localhost'
        11:39:35 );
        11:39:35 
        11:39:35 -- executing against localhost:21000
        11:39:35 describe formatted simple;
        11:39:35 
        11:39:35 -- executing against localhost:21000
        11:39:35 alter table simple set tblproperties ('kudu.master_addresses' = '127.0.0.1');
        11:39:35 
        11:39:35 -- executing against localhost:21000
        11:39:35 alter table simple set tblproperties ('kudu.master_addresses' = 'invalid_host');
        11:39:35 
        11:39:35 -- executing against localhost:21000
        11:39:35 alter table simple rename to simple_new;
        11:39:35 
        11:39:35 -- executing against localhost:21000
        11:39:35 select count(*) from simple_new;
        11:39:35 
        11:39:35 -- executing against localhost:21000
        11:39:35 create table tbl_to_alter (id int primary key, name string null, vali bigint not null)
        11:39:35   distribute by range (id) (partition 1 < values <= 10) stored as kudu
        11:39:35   tblproperties('kudu.table_name'='tbl_to_alter');
        11:39:35 
        11:39:35 -- executing against localhost:21000
        11:39:35 alter table tbl_to_alter add range partition 10 < values <= 20;
        11:39:35 
        11:39:35 -- executing against localhost:21000
        11:39:35 insert into tbl_to_alter values (15, 'name', 100);
        11:39:35 
        11:39:35 -- executing against localhost:21000
        11:39:35 select * from tbl_to_alter limit 1000;
        11:39:35 
        11:39:35 MainThread: Comparing QueryTestResults (expected vs actual):
        11:39:35 15,'name',100 != None
        11:39:35 Number of rows returned (expected vs actual): 1 != 0
        11:39:35  generated xml file: /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/ee_tests/results/TEST-impala-parallel.xml 
        11:39:35 =========================== short test summary info ============================
        
        Show
        kwho Michael Ho added a comment - Another failure: 11:39:35 =================================== FAILURES =================================== 11:39:35 TestKuduOperations.test_kudu_alter_table[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] 11:39:35 [gw3] linux2 -- Python 2.6.6 /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/bin/../infra/python/env/bin/python 11:39:35 query_test/test_kudu.py:62: in test_kudu_alter_table 11:39:35 self.run_test_case('QueryTest/kudu_alter', vector, use_db=unique_database) 11:39:35 common/impala_test_suite.py:368: in run_test_case 11:39:35 pytest.config.option.update_results, result_section='DML_RESULTS') 11:39:35 common/test_result_verifier.py:404: in verify_raw_results 11:39:35 VERIFIER_MAP[verifier](expected, actual) 11:39:35 common/test_result_verifier.py:231: in verify_query_result_is_equal 11:39:35 assert expected_results == actual_results 11:39:35 E assert Comparing QueryTestResults (expected vs actual): 11:39:35 E 15,'name',100 != None 11:39:35 E Number of rows returned (expected vs actual): 1 != 0 11:39:35 ---------------------------- Captured stderr setup ----------------------------- 11:39:35 SET sync_ddl=False; 11:39:35 -- executing against localhost:21000 11:39:35 DROP DATABASE IF EXISTS `test_kudu_alter_table_ddaaa05e` CASCADE; 11:39:35 11:39:35 SET sync_ddl=False; 11:39:35 -- executing against localhost:21000 11:39:35 CREATE DATABASE `test_kudu_alter_table_ddaaa05e`; 11:39:35 11:39:35 MainThread: Created database "test_kudu_alter_table_ddaaa05e" for test ID "query_test/test_kudu.py::TestKuduOperations::()::test_kudu_alter_table[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none]" 11:39:35 ----------------------------- Captured stderr call ----------------------------- 11:39:35 -- executing against localhost:21000 11:39:35 use test_kudu_alter_table_ddaaa05e; 11:39:35 11:39:35 SET disable_codegen=False; 11:39:35 SET abort_on_error=1; 11:39:35 SET exec_single_node_rows_threshold=0; 11:39:35 SET batch_size=0; 11:39:35 SET num_nodes=0; 11:39:35 -- executing against localhost:21000 11:39:35 create table simple (id int primary key, name string, valf float, vali bigint) 11:39:35 distribute by hash (id) into 3 buckets stored as kudu; 11:39:35 11:39:35 -- executing against localhost:21000 11:39:35 alter table simple set tblproperties ( 11:39:35 'kudu.master_addresses' = 'localhost' 11:39:35 ); 11:39:35 11:39:35 -- executing against localhost:21000 11:39:35 describe formatted simple; 11:39:35 11:39:35 -- executing against localhost:21000 11:39:35 alter table simple set tblproperties ('kudu.master_addresses' = '127.0.0.1'); 11:39:35 11:39:35 -- executing against localhost:21000 11:39:35 alter table simple set tblproperties ('kudu.master_addresses' = 'invalid_host'); 11:39:35 11:39:35 -- executing against localhost:21000 11:39:35 alter table simple rename to simple_new; 11:39:35 11:39:35 -- executing against localhost:21000 11:39:35 select count(*) from simple_new; 11:39:35 11:39:35 -- executing against localhost:21000 11:39:35 create table tbl_to_alter (id int primary key, name string null, vali bigint not null) 11:39:35 distribute by range (id) (partition 1 < values <= 10) stored as kudu 11:39:35 tblproperties('kudu.table_name'='tbl_to_alter'); 11:39:35 11:39:35 -- executing against localhost:21000 11:39:35 alter table tbl_to_alter add range partition 10 < values <= 20; 11:39:35 11:39:35 -- executing against localhost:21000 11:39:35 insert into tbl_to_alter values (15, 'name', 100); 11:39:35 11:39:35 -- executing against localhost:21000 11:39:35 select * from tbl_to_alter limit 1000; 11:39:35 11:39:35 MainThread: Comparing QueryTestResults (expected vs actual): 11:39:35 15,'name',100 != None 11:39:35 Number of rows returned (expected vs actual): 1 != 0 11:39:35 generated xml file: /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/ee_tests/results/TEST-impala-parallel.xml 11:39:35 =========================== short test summary info ============================
        Hide
        mjacobs Matthew Jacobs added a comment -

        The issue is that we set the kudu tbl name explicitly via tblproperty so it doesn't have the unique db name in there, and the tests are run concurrently in exhaustive so this test may end up running the multiple times (w/ diff parameters, e.g. disable_codegen) concurrently.

        This test needs to be run serially. I will post a fix.

        Show
        mjacobs Matthew Jacobs added a comment - The issue is that we set the kudu tbl name explicitly via tblproperty so it doesn't have the unique db name in there, and the tests are run concurrently in exhaustive so this test may end up running the multiple times (w/ diff parameters, e.g. disable_codegen) concurrently. This test needs to be run serially. I will post a fix.
        Hide
        mjacobs Matthew Jacobs added a comment -

        commit 48983b389366eb7f792f5e309fa3eb77b70990f6
        Author: Matthew Jacobs <mj@cloudera.com>
        Date: Thu Dec 1 12:59:47 2016 -0800

        IMPALA-4567: Fix test_kudu_alter_table exhaustive failures

        The issue is that we set the Kudu table name explicitly via
        tblproperty so it doesn't have the unique db name in the
        underlying Kudu name. Meanwhile, the tests are run
        concurrently in exhaustive so this test may end up running
        the multiple times (w/ different parameters, e.g.
        disable_codegen) concurrently. This test needs to be run
        serially.

        Change-Id: Ibca64d5567c24240606e454b052d130fcd0c3968
        Reviewed-on: http://gerrit.cloudera.org:8080/5312
        Reviewed-by: David Knupp <dknupp@cloudera.com>
        Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        mjacobs Matthew Jacobs added a comment - commit 48983b389366eb7f792f5e309fa3eb77b70990f6 Author: Matthew Jacobs <mj@cloudera.com> Date: Thu Dec 1 12:59:47 2016 -0800 IMPALA-4567 : Fix test_kudu_alter_table exhaustive failures The issue is that we set the Kudu table name explicitly via tblproperty so it doesn't have the unique db name in the underlying Kudu name. Meanwhile, the tests are run concurrently in exhaustive so this test may end up running the multiple times (w/ different parameters, e.g. disable_codegen) concurrently. This test needs to be run serially. Change-Id: Ibca64d5567c24240606e454b052d130fcd0c3968 Reviewed-on: http://gerrit.cloudera.org:8080/5312 Reviewed-by: David Knupp <dknupp@cloudera.com> Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com> Tested-by: Internal Jenkins

          People

          • Assignee:
            mjacobs Matthew Jacobs
            Reporter:
            kwho Michael Ho
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development