Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9995

ACID compaction tries to compact a single file

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 4.0.0, 4.0.0-alpha-1
    • Transactions
    • None

    Description

      Consider TestWorker.minorWithOpenInMiddle()

      since there is an open txnId=23, this doesn't have any meaningful minor compaction work to do. The system still tries to compact a single delta file for 21-22 id range, and effectively copies the file onto itself.

      This is 1. inefficient and 2. can potentially affect a reader.

      (from a real cluster)
      Suppose we start with

      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:03 /user/hive/warehouse/t/base_0000016
      -rw-r--r--   1 ekoifman staff        602 2016-06-09 16:03 /user/hive/warehouse/t/base_0000016/bucket_00000
      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:07 /user/hive/warehouse/t/base_0000017
      -rw-r--r--   1 ekoifman staff        588 2016-06-09 16:07 /user/hive/warehouse/t/base_0000017/bucket_00000
      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:07 /user/hive/warehouse/t/delta_0000017_0000017_0000
      -rw-r--r--   1 ekoifman staff        514 2016-06-09 16:06 /user/hive/warehouse/t/delta_0000017_0000017_0000/bucket_00000
      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:07 /user/hive/warehouse/t/delta_0000018_0000018_0000
      -rw-r--r--   1 ekoifman staff        612 2016-06-09 16:07 /user/hive/warehouse/t/delta_0000018_0000018_0000/bucket_00000
      

      then do alter table T compact 'minor';

      then we end up with

      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:07 /user/hive/warehouse/t/base_0000017
      -rw-r--r--   1 ekoifman staff        588 2016-06-09 16:07 /user/hive/warehouse/t/base_0000017/bucket_00000
      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:11 /user/hive/warehouse/t/delta_0000018_0000018
      -rw-r--r--   1 ekoifman staff        500 2016-06-09 16:11 /user/hive/warehouse/t/delta_0000018_0000018/bucket_00000
      drwxr-xr-x   - ekoifman staff          0 2016-06-09 16:07 /user/hive/warehouse/t/delta_0000018_0000018_0000
      -rw-r--r--   1 ekoifman staff        612 2016-06-09 16:07 /user/hive/warehouse/t/delta_0000018_0000018_0000/bucket_00000
      

      So compaction created a new dir /user/hive/warehouse/t/delta_0000018_0000018

      Attachments

        1. HIVE-9995.01.patch
          9 kB
          Eugene Koifman
        2. HIVE-9995.02.patch
          10 kB
          Denys Kuzmenko
        3. HIVE-9995.03.patch
          10 kB
          Denys Kuzmenko
        4. HIVE-9995.04.patch
          14 kB
          Denys Kuzmenko
        5. HIVE-9995.05.patch
          16 kB
          Denys Kuzmenko
        6. HIVE-9995.06.patch
          16 kB
          Denys Kuzmenko
        7. HIVE-9995.07.patch
          14 kB
          Denys Kuzmenko
        8. HIVE-9995.08.patch
          14 kB
          Denys Kuzmenko
        9. HIVE-9995.09.patch
          22 kB
          Denys Kuzmenko
        10. HIVE-9995.10.patch
          22 kB
          Denys Kuzmenko
        11. HIVE-9995.WIP.patch
          3 kB
          Eugene Koifman

        Issue Links

          Activity

            People

              dkuzmenko Denys Kuzmenko
              ekoifman Eugene Koifman
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: