Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8966

Delta files created by hive hcatalog streaming cannot be compacted

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 1.0.0
    • Component/s: HCatalog
    • Labels:
      None
    • Environment:

      hive

    • Release Note:
      Don't do compaction on the current delta if it has a file in bucket pattern but not compactable

      Description

      hive hcatalog streaming will also create a file like bucket_n_flush_length in each delta directory. Where "n" is the bucket number. But the compactor.CompactorMR think this file also needs to compact. However this file of course cannot be compacted, so compactor.CompactorMR will not continue to do the compaction.

      Did a test, after removed the bucket_n_flush_length file, then the "alter table partition compact" finished successfully. If don't delete that file, nothing will be compacted.
      This is probably a very severity bug. Both 0.13 and 0.14 have this issue

        Attachments

        1. HIVE-8966.patch
          2 kB
          Jihong Liu
        2. HIVE-8966.2.patch
          16 kB
          Alan Gates
        3. HIVE-8966.3.patch
          19 kB
          Alan Gates
        4. HIVE-8966.4.patch
          82 kB
          Alan Gates
        5. HIVE-8966.5.patch
          87 kB
          Alan Gates
        6. HIVE-8966.6.patch
          89 kB
          Alan Gates
        7. HIVE-8966-branch-1.patch
          101 kB
          Alan Gates

          Activity

            People

            • Assignee:
              alangates Alan Gates
              Reporter:
              jihongliu Jihong Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: