Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 4.0.0
    • Component/s: Transactions
    • Labels:
      None
    • Target Version/s:

      Description

      Have compactor open a transaction and run the job in that transaction.

      1. make compactor produced base/delta include this txn id in the folder name, e.g. base_7_c17 where 17 is the txnid.
      2. add CQ_TXN_ID bigint to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to record this txn id
      3. make sure AcidUtils.getAcidState() pays attention to this transaction on read and ignores this dir if this txn id is not committed in the current snapshot
        1. this means not only validWriteIdList but ValidTxnIdList should be passed along in config (if it isn't yet)
      4. once this is done, CompactorMR.createCompactorMarker() can be eliminated and AcidUtils.isValidBase modified accordingly
      5. modify Cleaner so that it doesn't clean old files until new file is visible to all readers

        Attachments

        1. HIVE-20823.14.patch
          224 kB
          Eugene Koifman
        2. HIVE-20823.13.patch
          230 kB
          Eugene Koifman
        3. HIVE-20823.12.patch
          225 kB
          Eugene Koifman
        4. HIVE-20823.11.patch
          218 kB
          Eugene Koifman
        5. HIVE-20823.11.patch
          217 kB
          Eugene Koifman
        6. HIVE-20823.10.patch
          186 kB
          Eugene Koifman
        7. HIVE-20823.09.patch
          184 kB
          Eugene Koifman
        8. HIVE-20823.08.patch
          184 kB
          Eugene Koifman
        9. HIVE-20823.07.patch
          142 kB
          Eugene Koifman
        10. HIVE-20823.05.patch
          144 kB
          Eugene Koifman
        11. HIVE-20823.04.patch
          116 kB
          Eugene Koifman
        12. HIVE-20823.03.patch
          90 kB
          Eugene Koifman
        13. HIVE-20823.01.patch
          4 kB
          Eugene Koifman

          Issue Links

            Activity

              People

              • Assignee:
                ekoifman Eugene Koifman
                Reporter:
                ekoifman Eugene Koifman
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: