Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9995

ACID compaction tries to compact a single file

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 4.0.0
    • Component/s: Transactions
    • Labels:
      None
    • Target Version/s:

      Description

      Consider TestWorker.minorWithOpenInMiddle()

      since there is an open txnId=23, this doesn't have any meaningful minor compaction work to do. The system still tries to compact a single delta file for 21-22 id range, and effectively copies the file onto itself.

      This is 1. inefficient and 2. can potentially affect a reader.

      (from a real cluster)
      Suppose we start with

      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:03 /user/hive/warehouse/t/base_0000016
      -rw-r--r--   1 ekoifman staff        602 2016-06-09 16:03 /user/hive/warehouse/t/base_0000016/bucket_00000
      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:07 /user/hive/warehouse/t/base_0000017
      -rw-r--r--   1 ekoifman staff        588 2016-06-09 16:07 /user/hive/warehouse/t/base_0000017/bucket_00000
      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:07 /user/hive/warehouse/t/delta_0000017_0000017_0000
      -rw-r--r--   1 ekoifman staff        514 2016-06-09 16:06 /user/hive/warehouse/t/delta_0000017_0000017_0000/bucket_00000
      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:07 /user/hive/warehouse/t/delta_0000018_0000018_0000
      -rw-r--r--   1 ekoifman staff        612 2016-06-09 16:07 /user/hive/warehouse/t/delta_0000018_0000018_0000/bucket_00000
      

      then do alter table T compact 'minor';

      then we end up with

      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:07 /user/hive/warehouse/t/base_0000017
      -rw-r--r--   1 ekoifman staff        588 2016-06-09 16:07 /user/hive/warehouse/t/base_0000017/bucket_00000
      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:11 /user/hive/warehouse/t/delta_0000018_0000018
      -rw-r--r--   1 ekoifman staff        500 2016-06-09 16:11 /user/hive/warehouse/t/delta_0000018_0000018/bucket_00000
      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:07 /user/hive/warehouse/t/delta_0000018_0000018_0000
      -rw-r--r--   1 ekoifman staff        612 2016-06-09 16:07 /user/hive/warehouse/t/delta_0000018_0000018_0000/bucket_00000
      

      So compaction created a new dir /user/hive/warehouse/t/delta_0000018_0000018

        Attachments

        1. HIVE-9995.10.patch
          22 kB
          Denys Kuzmenko
        2. HIVE-9995.09.patch
          22 kB
          Denys Kuzmenko
        3. HIVE-9995.08.patch
          14 kB
          Denys Kuzmenko
        4. HIVE-9995.07.patch
          14 kB
          Denys Kuzmenko
        5. HIVE-9995.06.patch
          16 kB
          Denys Kuzmenko
        6. HIVE-9995.05.patch
          16 kB
          Denys Kuzmenko
        7. HIVE-9995.04.patch
          14 kB
          Denys Kuzmenko
        8. HIVE-9995.03.patch
          10 kB
          Denys Kuzmenko
        9. HIVE-9995.02.patch
          10 kB
          Denys Kuzmenko
        10. HIVE-9995.01.patch
          9 kB
          Eugene Koifman
        11. HIVE-9995.WIP.patch
          3 kB
          Eugene Koifman

          Issue Links

            Activity

              People

              • Assignee:
                dkuzmenko Denys Kuzmenko
                Reporter:
                ekoifman Eugene Koifman
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: