• Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 4.0.0-alpha-1
    • Transactions
    • None


      Have compactor open a transaction and run the job in that transaction.

      1. make compactor produced base/delta include this txn id in the folder name, e.g. base_7_c17 where 17 is the txnid.
      2. add CQ_TXN_ID bigint to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to record this txn id
      3. make sure AcidUtils.getAcidState() pays attention to this transaction on read and ignores this dir if this txn id is not committed in the current snapshot
        1. this means not only validWriteIdList but ValidTxnIdList should be passed along in config (if it isn't yet)
      4. once this is done, CompactorMR.createCompactorMarker() can be eliminated and AcidUtils.isValidBase modified accordingly
      5. modify Cleaner so that it doesn't clean old files until new file is visible to all readers


        1. HIVE-20823.01.patch
          4 kB
          Eugene Koifman
        2. HIVE-20823.03.patch
          90 kB
          Eugene Koifman
        3. HIVE-20823.04.patch
          116 kB
          Eugene Koifman
        4. HIVE-20823.05.patch
          144 kB
          Eugene Koifman
        5. HIVE-20823.07.patch
          142 kB
          Eugene Koifman
        6. HIVE-20823.08.patch
          184 kB
          Eugene Koifman
        7. HIVE-20823.09.patch
          184 kB
          Eugene Koifman
        8. HIVE-20823.10.patch
          186 kB
          Eugene Koifman
        9. HIVE-20823.11.patch
          218 kB
          Eugene Koifman
        10. HIVE-20823.11.patch
          217 kB
          Eugene Koifman
        11. HIVE-20823.12.patch
          225 kB
          Eugene Koifman
        12. HIVE-20823.13.patch
          230 kB
          Eugene Koifman
        13. HIVE-20823.14.patch
          224 kB
          Eugene Koifman

        Issue Links



              ekoifman Eugene Koifman
              ekoifman Eugene Koifman
              0 Vote for this issue
              5 Start watching this issue