[IMPALA-7084] Partition doesn't exist after attempting to ALTER partition location to non-existing path

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 2.12.0
• Component/s: Catalog
• Labels: correctness ramp-up
• Epic Color: ghx-label-2
• Assignee: Unassigned
• Reporter: Gabor Kaszab
• Created: 28/May/18 11:53
• Updated: 31/May/18 09:25
• Resolved: 30/May/18 08:24

Description

 

 

create table test (a int) partitioned by (b int);
insert into test partition (b=1) values (1);

// create another partition that points to a different location.
alter table test add partition (b=2) location 'hdfs://localhost:20500/test-warehouse/test/b=2/';

// setting the first partition to a non existing location. This fails as expected. The error message is not exactly user friendly, though. Could have said "Invalid location or such".
alter table test partition (b=1) set location 'hdfs://localhost:20500/test-warehouse/test/b=5/';
Query: alter table test partition (b=1) set location 'hdfs://localhost:20500/test-warehouse/test/b=5/'
ERROR: TableLoadingException: Failed to load metadata for table: default.test
CAUSED BY: NullPointerException: null

// Setting the first partition to an existing location. This surprisingly fails as the partitions doesn't exist.
alter table test partition (b=1) set location 'hdfs://localhost:20500/test-warehouse/test/b=2/';
Query: alter table test partition (b=1) set location 'hdfs://localhost:20500/test-warehouse/test/b=1/'
ERROR: PartitionNotFoundException: Partition not found: TPartitionKeyValue(name:b, value:1)

// However, show partition displays b=1 partition as well.
show partitions test;
Query: show partitions test
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------+
| b | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location |
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------+
| 1 | -1 | 1 | 2B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://localhost:20500/test-warehouse/test/b=1 |
| 2 | -1 | 1 | 2B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://localhost:20500/test-warehouse/test/b=2 |
| Total | -1 | 2 | 4B | 0B | | | | |
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+------------------------------------------------+

// Invalidate metadata fixes this issue
invalidate metadata test;

// Now show partitions say that b=1 partition points to the non-existing location. (This is not a problem as the location would be
// created one data is put into that partition again.)
show partitions test;
Query: show partitions test
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------
--------------------------+
| b | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location 
|
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------
--------------------------+
| 1 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://localhost:2050
0/test-warehouse/test/b=5 |
| 2 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://localhost:2050
0/test-warehouse/test/b=2 |
| Total | -1 | 0 | 0B | 0B | | | | 
|
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------------------
--------------------------+

// After the invalidate metadata the partition is again visilbe for ALTER TABLE partition location commands.
alter table test partition (b=1) set location 'hdfs://localhost:20500/test-warehouse/test/b=2/';
Query: alter table test partition (b=1) set location 'hdfs://localhost:20500/test-warehouse/test/b=2/'
+--------------------------------------------------------+
| summary |
+--------------------------------------------------------+
| New location has been set for the specified partition. |
+--------------------------------------------------------+

// 

 

My expectation here would be that in case the first alter partition location command fails due to non-existing location then no state change should be done in the background. In addition seeing an existing partition as a non-existing one after a failed attempt to change some metadata is also undesired.