[HIVE-16299] MSCK REPAIR TABLE should enforce partition key order when adding unknown partitions

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 3.0.0
• Component/s: Metastore
• Assignee: Vihang Karajgaonkar
• Reporter: Dudu Markovitz
• Created: 26/Mar/17 20:16
• Updated: 22/May/18 23:58
• Resolved: 31/Mar/17 04:09

Description

https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java

static String getPartitionName(Path tablePath, Path partitionPath, Set<String> partCols)

------------------------------------------------------------------------------------

MSCK REPAIR validates that any sub-directory is in the format col=val and that there is indeed a partition column named "col".
However, there is no validation of the partition column location and as a result false partitions are being created and so are directories that match those partitions.

e.g. 1

hive> dfs -mkdir -p /user/hive/warehouse/t/a=1/a=2/a=3/b=4/c=5;
hive> create external table t (i int) partitioned by (a int,b int,c int) ;
OK
hive> msck repair table t;
OK
Partitions not in metastore: t:a=1/a=2/a=3/b=4/c=5
Repair: Added partition to metastore t:a=1/a=2/a=3/b=4/c=5
Time taken: 0.563 seconds, Fetched: 2 row(s)
hive> show partitions t;
OK
a=3/b=4/c=5
hive> dfs -ls -R /user/hive/warehouse/t;
drwxr-xr-x - cloudera supergroup 0 2017-03-26 13:07 /user/hive/warehouse/t/a=1
drwxr-xr-x - cloudera supergroup 0 2017-03-26 13:07 /user/hive/warehouse/t/a=1/a=2
drwxr-xr-x - cloudera supergroup 0 2017-03-26 13:07 /user/hive/warehouse/t/a=1/a=2/a=3
drwxr-xr-x - cloudera supergroup 0 2017-03-26 13:07 /user/hive/warehouse/t/a=1/a=2/a=3/b=4
drwxr-xr-x - cloudera supergroup 0 2017-03-26 13:07 /user/hive/warehouse/t/a=1/a=2/a=3/b=4/c=5
drwxrwxrwx - cloudera supergroup 0 2017-03-26 13:07 /user/hive/warehouse/t/a=3
drwxrwxrwx - cloudera supergroup 0 2017-03-26 13:07 /user/hive/warehouse/t/a=3/b=4
drwxrwxrwx - cloudera supergroup 0 2017-03-26 13:07 /user/hive/warehouse/t/a=3/b=4/c=5

e.g. 2
hive> dfs -mkdir -p /user/hive/warehouse/t/c=3/b=2/a=1;
hive> create external table t (i int) partitioned by (a int,b int,c int);
OK
hive> msck repair table t;
OK
Partitions not in metastore: t:c=3/b=2/a=1
Repair: Added partition to metastore t:c=3/b=2/a=1
Time taken: 0.512 seconds, Fetched: 2 row(s)
hive> show partitions t;
OK
a=1/b=2/c=3
hive> dfs -ls -R /user/hive/warehouse/t;
drwxrwxrwx - cloudera supergroup 0 2017-03-26 13:13 /user/hive/warehouse/t/a=1
drwxrwxrwx - cloudera supergroup 0 2017-03-26 13:13 /user/hive/warehouse/t/a=1/b=2
drwxrwxrwx - cloudera supergroup 0 2017-03-26 13:13 /user/hive/warehouse/t/a=1/b=2/c=3
drwxr-xr-x - cloudera supergroup 0 2017-03-26 13:12 /user/hive/warehouse/t/c=3
drwxr-xr-x - cloudera supergroup 0 2017-03-26 13:12 /user/hive/warehouse/t/c=3/b=2
drwxr-xr-x - cloudera supergroup 0 2017-03-26 13:12 /user/hive/warehouse/t/c=3/b=2/a=1

Issue Links

• is related to: HIVE-16347 HiveMetastoreChecker should skip listing partitions which are not valid when hive.msck.path.validation is set to skip or ignore
• links to: RB link