Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8966

Delta files created by hive hcatalog streaming cannot be compacted

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 0.14.0
    • 1.0.0
    • HCatalog
    • None
    • hive

    • Don't do compaction on the current delta if it has a file in bucket pattern but not compactable

    Description

      hive hcatalog streaming will also create a file like bucket_n_flush_length in each delta directory. Where "n" is the bucket number. But the compactor.CompactorMR think this file also needs to compact. However this file of course cannot be compacted, so compactor.CompactorMR will not continue to do the compaction.

      Did a test, after removed the bucket_n_flush_length file, then the "alter table partition compact" finished successfully. If don't delete that file, nothing will be compacted.
      This is probably a very severity bug. Both 0.13 and 0.14 have this issue

      Attachments

        1. HIVE-8966-branch-1.patch
          101 kB
          Alan Gates
        2. HIVE-8966.patch
          2 kB
          Jihong Liu
        3. HIVE-8966.6.patch
          89 kB
          Alan Gates
        4. HIVE-8966.5.patch
          87 kB
          Alan Gates
        5. HIVE-8966.4.patch
          82 kB
          Alan Gates
        6. HIVE-8966.3.patch
          19 kB
          Alan Gates
        7. HIVE-8966.2.patch
          16 kB
          Alan Gates

        Activity

          People

            gates Alan Gates
            jihongliu Jihong Liu
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: