Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18320

Support ACID Tables Replication

    XMLWordPrintableJSON

Details

    Description

      Currently, Full ACID and MM (Micro-Managed) tables are not supported for replication.
      Need to support it.

      Attachments

        1. Hive Replication_ ACID Tables.pdf
          443 kB
          Sankar Hariappan

        Issue Links

          1.
          Introduce WriteID per table rather than using global transaction ID Sub-task Closed Sankar Hariappan  
          2.
          Migrate existing ACID tables to use write id per table rather than global transaction id Sub-task Closed Sankar Hariappan  
          3.
          create/replicate open transaction event Sub-task Closed mahesh kumar behera  
          4.
          Replicate Commit Txn operation (without writes) Sub-task Closed mahesh kumar behera  
          5.
          Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL. Sub-task Closed Sankar Hariappan  
          6.
          Rename table impacts the ACID behavior as table names are not updated in meta-tables. Sub-task Closed Eugene Koifman  
          7.
          Need to┬áreplace transactionId with writeId in RecordIdentifier and other relevant contexts. Sub-task Closed Sankar Hariappan  
          8.
          Exchange partition should be disabled on ACID/Insert-only tables with per table write ID. Sub-task Closed Sankar Hariappan  
          9.
          ACID table scan through get_splits UDF doesn't receive ValidWriteIdList configuration. Sub-task Closed Sankar Hariappan  
          10.
          Correct methods and variables names which uses writeId instead of transactionId. Sub-task Closed Sankar Hariappan  
          11.
          ValidWriteIdList config should be defined on tables which has to collect stats after insert Sub-task Closed Sergey Shelukhin  
          12.
          Create/Replicate Open, Commit (without writes) and Abort Txn events Sub-task Closed mahesh kumar behera  
          13.
          ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction. Sub-task Closed Sankar Hariappan  
          14.
          Support change management for trashing data files from ACID tables. Sub-task Closed Anishek Agarwal  
          15.
          Support bootstrap replication of ACID tables Sub-task Closed Sankar Hariappan  
          16.
          Create/Replicate Allocate write-id event Sub-task Closed mahesh kumar behera  
          17.
          Replicate ACID/MM tables write operations. Sub-task Resolved mahesh kumar behera

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          18.
          Disable timeout of transactions opened by replication task at target cluster Sub-task Closed mahesh kumar behera

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          19.
          Last Repl ID set by bootstrap dump is incorrect and may cause data loss if have ACID/MM tables. Sub-task Resolved Sankar Hariappan  
          20.
          ptest optimization - Replicate ACID/MM tables write operations. Sub-task Resolved mahesh kumar behera

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          21.
          SQL Script changes for creating txn write notification in 3.2.0 files Sub-task Resolved mahesh kumar behera

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          22.
          Bootstrap repl dump with concurrent write and drop of ACID table makes target inconsistent. Sub-task Resolved Sankar Hariappan  
          23.
          Hive support bootstrap of ACID/MM tables on an existing policy. Sub-task Resolved Ashutosh Bapat

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 7h 10m
          24.
          Replicate Streaming ingestion with transactional batch size as 1. Sub-task Resolved Sankar Hariappan

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 40m

          Activity

            People

              sankarh Sankar Hariappan
              sankarh Sankar Hariappan
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 11.5h
                  11.5h