Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9092

Add support for creating external Kudu table

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • Impala 3.4.0
    • Impala 3.4.0
    • Frontend
    • None
    • ghx-label-12

    Description

      Hive changed behavior with HIVE-22158 so that only transactional tables are considered managed and all other are considered external. This means that a regular "create table" will result in an external table with table properties of 'TRANSLATED_TO_EXTERNAL'='TRUE', 'external.table.purge'='TRUE'. This breaks our tests that rely on "show create table", because the table is newly external and has extra table properties. For example:

      query_test/test_kudu.py:842: in test_primary_key_and_distribution
          db=cursor.conn.db_name, kudu_addr=KUDU_MASTER_HOSTS))
      query_test/test_kudu.py:824: in assert_show_create_equals
          assert cursor.fetchall()[0][0] == \
      E   assert "CREATE EXTER...='localhost')" == "CREATE TABLE ...='localhost')"
      E     - CREATE EXTERNAL TABLE testshowcreatetable_15312_ggn1hk.nvbpxfuxze
      E     ?        ---------
      E     + CREATE TABLE testshowcreatetable_15312_ggn1hk.nvbpxfuxze (
      E     ?                                                         ++
      E     +   c INT NOT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
      E     +   PRIMARY KEY (c)
      E     + )
      E     + PARTITION BY HASH (c) PARTITIONS 3
      E       STORED AS KUDU
      E     - TBLPROPERTIES ('TRANSLATED_TO_EXTERNAL'='TRUE', 'external.table.purge'='TRUE', 'kudu.master_addresses'='localhost')
      E     + TBLPROPERTIES ('kudu.master_addresses'='localhost')

      We need to decide on the right behavior for "show create table" and update the tests. 

      For Kudu tables, tables with TRANSLATED_TO_EXTERNAL=true and external.table.purge=TRUE should be equivalent to a non-external Kudu table, and we can just detect this case and generate the same SQL as before.

      Other cases may need new logic. I think it makes sense to also address other tests due to MANAGED vs EXTERNAL distinction or extra table properties with this JIRA. Here is a list of tests that seem to have this problem:

      metadata/test_ddl.py TestDdlStatements.test_create_alter_tbl_properties
      metadata/test_show_create_table.py *
      query_test/test_kudu.py TestShowCreateTable*
      org.apache.impala.catalog.CatalogTest.testCreateTableMetadata
      org.apache.impala.catalog.local.LocalCatalogTest.testKuduTable

      Attachments

        Issue Links

          Activity

            People

              vihangk1 Vihang Karajgaonkar
              joemcdonnell Joe McDonnell
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: