[HIVE-13703] "msck repair" on table with non-partition subdirectories reporting partitions not in metastore

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.14.0, 1.0.0, 1.2.1
• Component/s: Hive
• Assignee: Alina Abramova
• Reporter: Ana Gillan
• Created: 06/May/16 10:20
• Updated: 21/Sep/16 18:15
• Resolved: 21/Sep/16 18:15

Description

PROBLEM: Subdirectories created with UNION ALL are listed in show partitions output, but show up as Partitions not in metastore in msck repair output.

STEPS TO REPRODUCE: Table created from CTAS ... UNION ALL DDL

hive> msck repair table meter_001; 
OK 
Partitions not in metastore: meter_001:tech_datestamp=2016-03-09/1 meter_001:tech_datestamp=2016-03-09/2 meter_001:tech_datestamp=2016-03-10/1 meter_001:tech_datestamp=2016-03-10/2 meter_001:tech_datestamp=2016-03-11/1 meter_001:tech_datestamp=2016-03-11/2 meter_001:tech_datestamp=2016-03-12/1 meter_001:tech_datestamp=2016-03-12/2 meter_001:tech_datestamp=2016-03-13/1 meter_001:tech_datestamp=2016-03-13/2 meter_001:tech_datestamp=2016-03-14/1 meter_001:tech_datestamp=2016-03-14/2 meter_001:tech_datestamp=2016-03-15/1 meter_001:tech_datestamp=2016-03-15/2 meter_001:tech_datestamp=2016-03-16/1 meter_001:tech_datestamp=2016-03-16/2 meter_001:tech_datestamp=2016-03-17/1 meter_001:tech_datestamp=2016-03-17/2 meter_001:tech_datestamp=2016-03-18/1 meter_001:tech_datestamp=2016-03-18/2 meter_001:tech_datestamp=2016-03-19/1 meter_001:tech_datestamp=2016-03-19/2 meter_001:tech_datestamp=2016-03-20/1 meter_001:tech_datestamp=2016-03-20/2 meter_001:tech_datestamp=2016-03-21/1 meter_001:tech_datestamp=2016-03-21/2 meter_001:tech_datestamp=2016-03-22/1 meter_001:tech_datestamp=2016-03-22/2 meter_001:tech_datestamp=2016-03-23/1 meter_001:tech_datestamp=2016-03-23/2 meter_001:tech_datestamp=2016-03-24/1 meter_001:tech_datestamp=2016-03-24/2 meter_001:tech_datestamp=2016-03-25/1 meter_001:tech_datestamp=2016-03-25/2 meter_001:tech_datestamp=2016-03-26/1 meter_001:tech_datestamp=2016-03-26/2 meter_001:tech_datestamp=2016-03-27/1 meter_001:tech_datestamp=2016-03-27/2 meter_001:tech_datestamp=2016-03-28/1 meter_001:tech_datestamp=2016-03-28/2 meter_001:tech_datestamp=2016-03-29/1 meter_001:tech_datestamp=2016-03-29/2 meter_001:tech_datestamp=2016-03-30/1 meter_001:tech_datestamp=2016-03-30/2 meter_001:tech_datestamp=2016-03-31/1 meter_001:tech_datestamp=2016-03-31/2 meter_001:tech_datestamp=2016-04-01/1 meter_001:tech_datestamp=2016-04-01/2 meter_001:tech_datestamp=2016-04-02/1 meter_001:tech_datestamp=2016-04-02/2 meter_001:tech_datestamp=2016-04-03/1 meter_001:tech_datestamp=2016-04-03/2 meter_001:tech_datestamp=2016-04-04/1 meter_001:tech_datestamp=2016-04-04/2 meter_001:tech_datestamp=2016-04-05/1 meter_001:tech_datestamp=2016-04-05/2 meter_001:tech_datestamp=2016-04-06/1 meter_001:tech_datestamp=2016-04-06/2 
Time taken: 15.996 seconds, Fetched: 1 row(s) 

hive> show partitions meter_001; 
OK 
tech_datestamp=2016-03-09 
tech_datestamp=2016-03-10 
tech_datestamp=2016-03-11 
tech_datestamp=2016-03-12 
tech_datestamp=2016-03-13 
tech_datestamp=2016-03-14 
tech_datestamp=2016-03-15 
tech_datestamp=2016-03-16 
tech_datestamp=2016-03-17 
tech_datestamp=2016-03-18 
tech_datestamp=2016-03-19 
tech_datestamp=2016-03-20 
tech_datestamp=2016-03-21 
tech_datestamp=2016-03-22 
tech_datestamp=2016-03-23 
tech_datestamp=2016-03-24 
tech_datestamp=2016-03-25 
tech_datestamp=2016-03-26 
tech_datestamp=2016-03-27 
tech_datestamp=2016-03-28 
tech_datestamp=2016-03-29 
tech_datestamp=2016-03-30 
tech_datestamp=2016-03-31 
tech_datestamp=2016-04-01 
tech_datestamp=2016-04-02 
tech_datestamp=2016-04-03 
tech_datestamp=2016-04-04 
tech_datestamp=2016-04-05 
tech_datestamp=2016-04-06 
Time taken: 0.417 seconds, Fetched: 29 row(s) 

Ideally msck repair should ignore subdirectory if that additional partition column doesn't exist.

Issue Links

• Is contained by: HIVE-14511 Improve MSCK for partitioned table to deal with special cases
• relates to: HIVE-13425 Fix partition addition in MSCK REPAIR TABLE command