Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3986

Python hive-client may fail silently while dropping partitions

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • Impala 2.6.0
    • Impala 2.7.0
    • Infrastructure
    • None

    Description

      http://sandbox.jenkins.cloudera.com/job/impala-asf-master-core-s3/38

      sailesh - I picked you thinking you might have an idea what’s going on here, as this seems to affect S3; feel free to find another person or assign back to me if you're swamped.

      Some tests in QueryTest/insert.test perform pre-test clean sections prior to executing the actual test. The cleanup for this one (line 822ff) fails:

      ---- QUERY
SET S3_SKIP_INSERT_STAGING=false;
# fully dynamic partition insert$TABLE, check partition creation
insert into table alltypesinsert
partition (year, month)
select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col,
float_col, double_col, date_string_col, string_col, timestamp_col, year, month
from alltypessmall
---- SETUP
DROP PARTITIONS alltypesinsert
---- RESULTS
year=2009/month=1/: 25
year=2009/month=2/: 25
year=2009/month=3/: 25
year=2009/month=4/: 25
====

      The full error is the following. It looks like file for the partition year=2009/month=2 is missing or has already been deleted.

      sailesh, do you have any idea how this could happen? Do we need to run invalidate metadata before cleaning the partitions?

      04:43:58.475 =================================== FAILURES ===================================
04:43:58.475  TestInsertQueries.test_insert[exec_option: {'batch_size': 0, 'num_nodes': 0, 'sync_ddl': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none] 
04:43:58.475 query_test/test_insert.py:111: in test_insert
04:43:58.475     multiple_impalad=vector.get_value('exec_option')['sync_ddl'] == 1)
04:43:58.475 common/impala_test_suite.py:276: in run_test_case
04:43:58.475     self.execute_test_case_setup(test_section['SETUP'], table_format_info)
04:43:58.475 common/impala_test_suite.py:368: in execute_test_case_setup
04:43:58.475     self.__drop_partitions(db_name, table_name)
04:43:58.475 common/impala_test_suite.py:493: in __drop_partitions
04:43:58.475     self.hive_client.drop_partition_by_name(db_name, table_name, partition, True)
04:43:58.475 ../shell/gen-py/hive_metastore/ThriftHiveMetastore.py:2502: in drop_partition_by_name
04:43:58.475     return self.recv_drop_partition_by_name()
04:43:58.475 ../shell/gen-py/hive_metastore/ThriftHiveMetastore.py:2530: in recv_drop_partition_by_name
04:43:58.475     raise result.o2
04:43:58.475 E   MetaException: MetaException(_message='No such file or directory: s3a://impala-cdh5-trunk/test-warehouse/functional.db/alltypesinsert/year=2009/month=2')
04:43:58.475 ---------------------------- Captured stderr setup -----------------------------
04:43:58.475 -- connecting to: localhost:21000
04:43:58.475 ----------------------------- Captured stderr call -----------------------------
04:43:58.475 -- executing against localhost:21000
04:43:58.475 use functional;
04:43:58.475 
04:43:58.475 SET batch_size=0;
04:43:58.475 SET num_nodes=0;
04:43:58.475 SET sync_ddl=0;
04:43:58.475 SET disable_codegen=False;
04:43:58.475 SET abort_on_error=1;
04:43:58.475 SET exec_single_node_rows_threshold=0;
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. alltypesnopart_insert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 insert overwrite table alltypesnopart_insert
04:43:58.475 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col,
04:43:58.475 float_col, double_col, date_string_col, string_col, timestamp_col
04:43:58.475 from alltypessmall
04:43:58.475 where year=2009 and month=04;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col,
04:43:58.475 double_col, date_string_col, string_col
04:43:58.475 from alltypesnopart_insert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. alltypesnopart_insert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 insert into table alltypesnopart_insert
04:43:58.475 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col,
04:43:58.475 float_col, double_col, date_string_col, string_col, timestamp_col
04:43:58.475 from alltypessmall
04:43:58.475 where year=2009 and month=04;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 select count(*) from alltypesnopart_insert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. alltypesinsert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. alltypesinsert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 insert overwrite table alltypesinsert
04:43:58.475 partition (year=2009, month=4)
04:43:58.475 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col,
04:43:58.475 float_col, double_col, date_string_col, string_col, timestamp_col
04:43:58.475 from alltypessmall
04:43:58.475 where year=2009 and month=4;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col,
04:43:58.475 double_col, date_string_col, string_col
04:43:58.475 from alltypesinsert
04:43:58.475 where year=2009 and month=4;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. alltypesinsert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 insert into table alltypesinsert
04:43:58.475 partition (year=2009, month=4)
04:43:58.475 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col,
04:43:58.475 float_col, double_col, date_string_col, string_col, timestamp_col
04:43:58.475 from alltypessmall
04:43:58.475 where year=2009 and month=4;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 select count(*) from alltypesinsert
04:43:58.475 where year=2009 and month=4;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. alltypesinsert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 insert overwrite table alltypesinsert
04:43:58.475 partition (year=2009, month)
04:43:58.475 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col,
04:43:58.475 float_col, double_col, date_string_col, string_col, timestamp_col, month
04:43:58.475 from alltypessmall
04:43:58.475 where year=2009 and month>1 and month<=4;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col,
04:43:58.475 double_col, date_string_col, string_col
04:43:58.475 from alltypesinsert
04:43:58.475 where year=2009 and month>1 and month<=4;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. alltypesinsert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 insert into table alltypesinsert
04:43:58.475 partition (year=2009, month)
04:43:58.475 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col,
04:43:58.475 float_col, double_col, date_string_col, string_col, timestamp_col, month
04:43:58.475 from alltypessmall
04:43:58.475 where year=2009 and month>=1 and month<4;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 select count(id) from alltypesinsert
04:43:58.475 where year=2009 and month>=1 and month<4;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. alltypesinsert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 insert overwrite table alltypesinsert
04:43:58.475 partition (year, month)
04:43:58.475 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col,
04:43:58.475 float_col, double_col, date_string_col, string_col, timestamp_col, year, month
04:43:58.475 from alltypessmall;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col,
04:43:58.475 double_col, date_string_col, string_col
04:43:58.475 from alltypesinsert
04:43:58.475 where year=2009 and month>=1 and month<=4;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. alltypesinsert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 insert into table alltypesinsert
04:43:58.475 partition (year, month)
04:43:58.475 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col,
04:43:58.475 float_col, double_col, date_string_col, string_col, timestamp_col, year, month
04:43:58.475 from alltypessmall;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 select count(timestamp_col) from alltypesinsert
04:43:58.475 where year=2009 and month>=1 and month<=4;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. alltypesinsert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 insert into alltypesinsert partition(year, month) select * from alltypessmall;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 select count(*) from alltypesinsert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. insert_string_partitioned;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. insert_string_partitioned;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 INSERT INTO TABLE insert_string_partitioned PARTITION(s2="/\%.") SELECT "value" FROM alltypessmall LIMIT 1;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 SELECT * FROM insert_string_partitioned;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 SELECT * FROM insert_string_partitioned WHERE s2 = "/\%.";
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. insert_string_partitioned;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 INSERT INTO TABLE insert_string_partitioned PARTITION(s2="_.~ +")
04:43:58.475 SELECT "value" FROM alltypessmall LIMIT 1;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 SELECT * FROM insert_string_partitioned;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. insert_string_partitioned;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 INSERT INTO TABLE insert_string_partitioned PARTITION(s2='')
04:43:58.475 SELECT "value1" FROM alltypessmall LIMIT 1;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 INSERT INTO TABLE insert_string_partitioned PARTITION(s2)
04:43:58.475 SELECT "value2","" FROM alltypessmall LIMIT 1;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 INSERT INTO TABLE insert_string_partitioned PARTITION(s2=NULL)
04:43:58.475 SELECT "value3" FROM alltypessmall LIMIT 1;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 INSERT INTO TABLE insert_string_partitioned PARTITION(s2)
04:43:58.475 SELECT "value4",NULL FROM alltypessmall LIMIT 1;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 SELECT * FROM insert_string_partitioned WHERE s2 = "";
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 SELECT s1, s2 FROM insert_string_partitioned where s2 is NULL
04:43:58.475 order by s1 limit 10;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. alltypesinsert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 insert into alltypesinsert partition(year=cast(100*20+10 as int), month=cast(2*2 as int))
04:43:58.475 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col,
04:43:58.475 double_col, date_string_col, string_col, timestamp_col from alltypessmall where month = 4;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. alltypesinsert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 insert into alltypesinsert
04:43:58.475 partition(year=2010, month=4)
04:43:58.475 select 100, false, 1, 1, 1, 10,
04:43:58.475 10.0, 10.0, "02/01/09", "1", cast("2009-02-01 00:01:00" as timestamp);
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 insert into table alltypesinsert
04:43:58.475 partition (year, month)
04:43:58.475 select 200, true, 2, 2, 2, 20,
04:43:58.475 20.0, 20.0, "02/01/09", "1", cast("2009-02-01 00:02:00" as timestamp), 2010, 4;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 select * from alltypesinsert order by id limit 2;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 invalidate metadata functional. alltypesinsert;
04:43:58.475 
04:43:58.475 -- executing against localhost:21000
04:43:58.475 insert into alltypesinsert
04:43:58.475 partition(year=2010, month=4) values
04:43:58.475 (1, false, 1, 1, 1, 10, NULL, 10.0, "02/01/09", "1", cast("2009-02-01 00:01:00" as timestamp)),
04:43:58.475 (2, true, 2, NULL, 2, 20, 20.0, 20.0, "02/02/09", "2", NULL),
04:43:58.475 (3, false, 3, 3, 3, 30, 30.0, 30.0, "02/03/09", NULL, cast("2009-02-03 00:01:00" as timestamp));
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 insert into table alltypesinsert
04:43:58.476 partition (year, month) values
04:43:58.476 (4, true, 4, 4, 4, 40, NULL, 40.0, "02/04/09", "4", cast("2009-02-04 00:01:00" as timestamp), 2010, 4),
04:43:58.476 (5, false, 5, NULL, 5, 50, 50.0, 50.0, "02/05/09", "5", NULL, 2010, 8),
04:43:58.476 (6, true, 6, 6, 6, 60, 60.0, 60.0, "02/06/09", NULL, cast("2009-02-06 00:01:00" as timestamp), 2010, 8);
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 select * from alltypesinsert order by id limit 10;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 invalidate metadata functional. alltypesinsert;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 with t1 as (select * from alltypestiny)
04:43:58.476 insert into alltypesinsert partition(year, month) select * from t1;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 invalidate metadata functional. alltypesinsert;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 with t1 as (select * from alltypestiny)
04:43:58.476 insert into alltypesinsert partition(year, month)
04:43:58.476 with t2 as (select * from alltypestiny)
04:43:58.476 select * from t1 union all select * from t2;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 insert overwrite table alltypesnopart_insert
04:43:58.476 select NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
04:43:58.476 from alltypessmall limit 10;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col,
04:43:58.476 float_col, double_col, date_string_col, string_col, timestamp_col
04:43:58.476 from alltypesnopart_insert;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 invalidate metadata functional. alltypesinsert;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 insert overwrite alltypesnopart_insert(float_col, double_col)
04:43:58.476 values(CAST(1/0 AS FLOAT), 1/0), (CAST(-1/0 AS FLOAT), -1/0),
04:43:58.476       (CAST(0/0 AS FLOAT), 0/0), (CAST(-sqrt(-1) AS FLOAT), -sqrt(-1));
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 select CAST(float_col AS string), CAST(double_col AS string) from alltypesnopart_insert
04:43:58.476 order by float_col, double_col limit 10;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 insert overwrite table alltypesinsert
04:43:58.476 partition (year, month)
04:43:58.476 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col,
04:43:58.476 float_col, double_col, date_string_col, string_col, timestamp_col, year, month
04:43:58.476 from alltypessmall limit 0;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 select count(*) from alltypesinsert;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 insert overwrite table alltypesnopart_insert
04:43:58.476 select 1, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
04:43:58.476 from alltypessmall limit 0;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 select count(*) from alltypesnopart_insert;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 insert overwrite table alltypesinsert
04:43:58.476 partition (year, month)
04:43:58.476 select id, bool_col, tinyint_col, smallint_col, int_col,
04:43:58.476 sum(bigint_col) over(order by string_col),
04:43:58.476 float_col, double_col, date_string_col, string_col, timestamp_col,
04:43:58.476 int_col, month
04:43:58.476 from functional.alltypestiny;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 truncate table alltypesinsert;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 
04:43:58.476 insert overwrite table alltypesinsert
04:43:58.476 partition (year=2015, month=8)
04:43:58.476 select a.id, a.bool_col, a.tinyint_col, a.smallint_col, a.int_col, a.bigint_col,
04:43:58.476   b.float_col, b.double_col, b.date_string_col, b.string_col, b.timestamp_col
04:43:58.476 from
04:43:58.476 (select id, false bool_col, 1 tinyint_col, 2 smallint_col, 3 int_col, 4 bigint_col
04:43:58.476  from functional.alltypestiny where id between 0 and 2) a
04:43:58.476 full outer join
04:43:58.476 (select id, 5 float_col, 6 double_col, "s1" date_string_col, "s2" string_col,
04:43:58.476    cast("2009-02-06 00:01:00" as timestamp) timestamp_col
04:43:58.476  from functional.alltypestiny where id between 1 and 3) b
04:43:58.476 on (a.id = b.id);
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 select * from alltypesinsert;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 invalidate metadata functional. table_with_header_insert;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 insert into table_with_header_insert values (1), (2), (3), (4);
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 select * from table_with_header_insert;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 invalidate metadata functional. alltypesinsert;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 invalidate metadata functional. alltypesinsert;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 SET S3_SKIP_INSERT_STAGING=false;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 
04:43:58.476 insert overwrite table alltypesinsert
04:43:58.476 partition (year=2009, month=4)
04:43:58.476 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col,
04:43:58.476 float_col, double_col, date_string_col, string_col, timestamp_col
04:43:58.476 from alltypessmall
04:43:58.476 where year=2009 and month=4;
04:43:58.476 
04:43:58.476 -- executing against localhost:21000
04:43:58.476 select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col,
04:43:58.476 double_col, date_string_col, string_col
04:43:58.476 from alltypesinsert
04:43:58.476 where year=2009 and month=4;

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-7266 test_insert failure: Unable to drop partition on s3

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              sailesh Sailesh Mukil
              lv Lars Volker
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: