Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20699

Query based compactor for full CRUD Acid tables

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: Transactions
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Currently the Acid compactor is implemented as generated MR job (CompactorMR.java).

      It could also be expressed as a Hive query that reads from a given partition and writes data back to the same partition. This will merge the deltas and 'apply' the delete events. The simplest would be to just use Insert Overwrite but that will change all ROW__IDs which we don't want.

      Need to implement this in a way that preserves ROW__IDs and creates a new base_x directory to handle Major compaction.

      Minor compaction will be investigated separately.

        Attachments

        1. HIVE-20699.9.patch
          61 kB
          Vaibhav Gumashta
        2. HIVE-20699.8.patch
          61 kB
          Vaibhav Gumashta
        3. HIVE-20699.7.patch
          59 kB
          Vaibhav Gumashta
        4. HIVE-20699.6.patch
          59 kB
          Vaibhav Gumashta
        5. HIVE-20699.5.patch
          59 kB
          Vaibhav Gumashta
        6. HIVE-20699.4.patch
          58 kB
          Vaibhav Gumashta
        7. HIVE-20699.3.patch
          58 kB
          Vaibhav Gumashta
        8. HIVE-20699.2.patch
          41 kB
          Vaibhav Gumashta
        9. HIVE-20699.11.patch
          66 kB
          Vaibhav Gumashta
        10. HIVE-20699.11.patch
          66 kB
          Vaibhav Gumashta
        11. HIVE-20699.10.patch
          66 kB
          Vaibhav Gumashta
        12. HIVE-20699.1.patch
          23 kB
          Vaibhav Gumashta
        13. HIVE-20699.1.patch
          25 kB
          Vaibhav Gumashta

          Issue Links

            Activity

              People

              • Assignee:
                vgumashta Vaibhav Gumashta
                Reporter:
                ekoifman Eugene Koifman
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: