Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 2.11.0
-
None
-
ghx-label-9
Description
When a table has a partition with a special character in it's name, then the HDFS directory contains a percent sign (due to an escaped/UrlEncoded sequence). This is not decoded and compared properly when running alter table recover partitions. This creates new, unneeded partitions on each execution.
The steps to reproduce/demonstrate the issue:
[nightly-2:21000] > CREATE TABLE tbl_with_partition(col1 string) partitioned by (p string); Query: CREATE TABLE tbl_with_partition(col1 string) partitioned by (p string) Fetched 0 row(s) in 1.08s [nightly-2:21000] > ALTER TABLE tbl_with_partition add partition (p='100%'); Query: ALTER TABLE tbl_with_partition add partition (p='100%') Fetched 0 row(s) in 5.72s [nightly-2:21000] > show partitions tbl_with_partition; Query: show partitions tbl_with_partition +-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------------------+ | p | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location | +-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------------------+ | 100% | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%25 | | Total | -1 | 0 | 0B | 0B | | | | | +-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------------------+ Fetched 2 row(s) in 0.02s [nightly-2:21000] > ALTER TABLE tbl_with_partition recover partitions; Query: ALTER TABLE tbl_with_partition recover partitions Fetched 0 row(s) in 0.29s [nightly-2:21000] > show partitions tbl_with_partition; Query: show partitions tbl_with_partition +--------+-------+--------+------+--------------+-------------------+--------+-------------------+--------------------------------------------------------------+ | p | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location | +--------+-------+--------+------+--------------+-------------------+--------+-------------------+--------------------------------------------------------------+ | 100% | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%25 | | 100%25 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%2525 | | Total | -1 | 0 | 0B | 0B | | | | | +--------+-------+--------+------+--------------+-------------------+--------+-------------------+--------------------------------------------------------------+ Fetched 3 row(s) in 0.02s [nightly-2:21000] > ALTER TABLE tbl_with_partition recover partitions; Query: ALTER TABLE tbl_with_partition recover partitions Fetched 0 row(s) in 0.27s [nightly-2:21000] > show partitions tbl_with_partition; Query: show partitions tbl_with_partition +----------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------------------------------------------------+ | p | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location | +----------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------------------------------------------------+ | 100% | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%25 | | 100%25 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%2525 | | 100%2525 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://ns1/user/hive/warehouse/tbl_with_partition/p=100%252525 | | Total | -1 | 0 | 0B | 0B | | | | | +----------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------------------------------------------------+ Fetched 4 row(s) in 0.02s