Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13703

"msck repair" on table with non-partition subdirectories reporting partitions not in metastore

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.14.0, 1.0.0, 1.2.1
    • Fix Version/s: None
    • Component/s: Hive
    • Labels:
      None

      Description

      PROBLEM: Subdirectories created with UNION ALL are listed in show partitions output, but show up as Partitions not in metastore in msck repair output.

      STEPS TO REPRODUCE: Table created from CTAS ... UNION ALL DDL

      hive> msck repair table meter_001; 
      OK 
      Partitions not in metastore: meter_001:tech_datestamp=2016-03-09/1 meter_001:tech_datestamp=2016-03-09/2 meter_001:tech_datestamp=2016-03-10/1 meter_001:tech_datestamp=2016-03-10/2 meter_001:tech_datestamp=2016-03-11/1 meter_001:tech_datestamp=2016-03-11/2 meter_001:tech_datestamp=2016-03-12/1 meter_001:tech_datestamp=2016-03-12/2 meter_001:tech_datestamp=2016-03-13/1 meter_001:tech_datestamp=2016-03-13/2 meter_001:tech_datestamp=2016-03-14/1 meter_001:tech_datestamp=2016-03-14/2 meter_001:tech_datestamp=2016-03-15/1 meter_001:tech_datestamp=2016-03-15/2 meter_001:tech_datestamp=2016-03-16/1 meter_001:tech_datestamp=2016-03-16/2 meter_001:tech_datestamp=2016-03-17/1 meter_001:tech_datestamp=2016-03-17/2 meter_001:tech_datestamp=2016-03-18/1 meter_001:tech_datestamp=2016-03-18/2 meter_001:tech_datestamp=2016-03-19/1 meter_001:tech_datestamp=2016-03-19/2 meter_001:tech_datestamp=2016-03-20/1 meter_001:tech_datestamp=2016-03-20/2 meter_001:tech_datestamp=2016-03-21/1 meter_001:tech_datestamp=2016-03-21/2 meter_001:tech_datestamp=2016-03-22/1 meter_001:tech_datestamp=2016-03-22/2 meter_001:tech_datestamp=2016-03-23/1 meter_001:tech_datestamp=2016-03-23/2 meter_001:tech_datestamp=2016-03-24/1 meter_001:tech_datestamp=2016-03-24/2 meter_001:tech_datestamp=2016-03-25/1 meter_001:tech_datestamp=2016-03-25/2 meter_001:tech_datestamp=2016-03-26/1 meter_001:tech_datestamp=2016-03-26/2 meter_001:tech_datestamp=2016-03-27/1 meter_001:tech_datestamp=2016-03-27/2 meter_001:tech_datestamp=2016-03-28/1 meter_001:tech_datestamp=2016-03-28/2 meter_001:tech_datestamp=2016-03-29/1 meter_001:tech_datestamp=2016-03-29/2 meter_001:tech_datestamp=2016-03-30/1 meter_001:tech_datestamp=2016-03-30/2 meter_001:tech_datestamp=2016-03-31/1 meter_001:tech_datestamp=2016-03-31/2 meter_001:tech_datestamp=2016-04-01/1 meter_001:tech_datestamp=2016-04-01/2 meter_001:tech_datestamp=2016-04-02/1 meter_001:tech_datestamp=2016-04-02/2 meter_001:tech_datestamp=2016-04-03/1 meter_001:tech_datestamp=2016-04-03/2 meter_001:tech_datestamp=2016-04-04/1 meter_001:tech_datestamp=2016-04-04/2 meter_001:tech_datestamp=2016-04-05/1 meter_001:tech_datestamp=2016-04-05/2 meter_001:tech_datestamp=2016-04-06/1 meter_001:tech_datestamp=2016-04-06/2 
      Time taken: 15.996 seconds, Fetched: 1 row(s) 
      
      hive> show partitions meter_001; 
      OK 
      tech_datestamp=2016-03-09 
      tech_datestamp=2016-03-10 
      tech_datestamp=2016-03-11 
      tech_datestamp=2016-03-12 
      tech_datestamp=2016-03-13 
      tech_datestamp=2016-03-14 
      tech_datestamp=2016-03-15 
      tech_datestamp=2016-03-16 
      tech_datestamp=2016-03-17 
      tech_datestamp=2016-03-18 
      tech_datestamp=2016-03-19 
      tech_datestamp=2016-03-20 
      tech_datestamp=2016-03-21 
      tech_datestamp=2016-03-22 
      tech_datestamp=2016-03-23 
      tech_datestamp=2016-03-24 
      tech_datestamp=2016-03-25 
      tech_datestamp=2016-03-26 
      tech_datestamp=2016-03-27 
      tech_datestamp=2016-03-28 
      tech_datestamp=2016-03-29 
      tech_datestamp=2016-03-30 
      tech_datestamp=2016-03-31 
      tech_datestamp=2016-04-01 
      tech_datestamp=2016-04-02 
      tech_datestamp=2016-04-03 
      tech_datestamp=2016-04-04 
      tech_datestamp=2016-04-05 
      tech_datestamp=2016-04-06 
      Time taken: 0.417 seconds, Fetched: 29 row(s) 
      

      Ideally msck repair should ignore subdirectory if that additional partition column doesn't exist.

        Attachments

        1. HIVE-13703.patch
          1 kB
          Alina Abramova

          Issue Links

            Activity

              People

              • Assignee:
                alina.abramova Alina Abramova
                Reporter:
                agillan Ana Gillan
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: