Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6619

Alter table recover partitions creates unneeded partitions when faces percent sign

    Details

    • Epic Color:
      ghx-label-9

      Description

      When a table has a partition with a special character in it's name, then the HDFS directory contains a percent sign (due to an escaped/UrlEncoded sequence). This is not decoded and compared properly when running alter table recover partitions. This creates new, unneeded partitions on each execution.

      The steps to reproduce/demonstrate the issue:

      [nightly-2:21000] > CREATE TABLE tbl_with_partition(col1 string) partitioned by (p string);
      Query: CREATE TABLE tbl_with_partition(col1 string) partitioned by (p string)
      Fetched 0 row(s) in 1.08s
      [nightly-2:21000] > ALTER TABLE tbl_with_partition add partition (p='100%');
      Query: ALTER TABLE tbl_with_partition add partition (p='100%')
      Fetched 0 row(s) in 5.72s
      [nightly-2:21000] > show partitions tbl_with_partition;
      Query: show partitions tbl_with_partition
      +-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------------------+
      | p | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location |
      +-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------------------+
      | 100% | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%25 |
      | Total | -1 | 0 | 0B | 0B | | | | |
      +-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------------------+
      Fetched 2 row(s) in 0.02s
      [nightly-2:21000] > ALTER TABLE tbl_with_partition recover partitions;
      Query: ALTER TABLE tbl_with_partition recover partitions
      Fetched 0 row(s) in 0.29s
      [nightly-2:21000] > show partitions tbl_with_partition;
      Query: show partitions tbl_with_partition
      +--------+-------+--------+------+--------------+-------------------+--------+-------------------+--------------------------------------------------------------+
      | p | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location |
      +--------+-------+--------+------+--------------+-------------------+--------+-------------------+--------------------------------------------------------------+
      | 100% | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%25 |
      | 100%25 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%2525 |
      | Total | -1 | 0 | 0B | 0B | | | | |
      +--------+-------+--------+------+--------------+-------------------+--------+-------------------+--------------------------------------------------------------+
      Fetched 3 row(s) in 0.02s
      [nightly-2:21000] > ALTER TABLE tbl_with_partition recover partitions;
      Query: ALTER TABLE tbl_with_partition recover partitions
      Fetched 0 row(s) in 0.27s
      [nightly-2:21000] > show partitions tbl_with_partition;
      Query: show partitions tbl_with_partition
      +----------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------------------------------------------------+
      | p | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location |
      +----------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------------------------------------------------+
      | 100% | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%25 |
      | 100%25 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%2525 |
      | 100%2525 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%252525 |
      | Total | -1 | 0 | 0B | 0B | | | | |
      +----------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------------------------------------------------+
      Fetched 4 row(s) in 0.02s

        Attachments

          Activity

            People

            • Assignee:
              fredyw Fredy Wijaya
              Reporter:
              mszurap Miklos Szurap
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: