Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21158

Perform update split early

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.0.0
    • None
    • Transactions
    • None

    Description

      Currently Acid 2.0 does U=D+I in the OrcRecordUpdater. This means that all Updates (wide rows) are shuffled AND sorted.
      We could modify the the multi-insert statement which results from Merge statement so that instead of having one of the legs represent Update, we create 2 legs - 1 representing Delete of original row and 1 representing Insert of the new version.
      Delete events are very small so sorting them is cheap. The Insert are written to disk in a sorted way by virtue of how ROW__IDs are generated.

      Exactly the same idea applies to regular Update statement.

      Note that the U=D+I in OrcRecordUpdater needs to be kept to keep Streaming Mutate API working on 2.0.

      This requires that TxnHandler flags 2 Deletes as a conflict - it doesn't currently

      Incidentally, 2.0 + early split allows updating all columns including bucketing and partition columns

      What is lock acquisition based on? Need to make sure that conflict detection (write set tracking) still works

      So we want to transform

      update T set B = 7 where A=1
      

      into

      from T
      insert into T select ROW__ID where a = 1 SORT BY ROW__ID
      insert into T select a, 7 where a = 1
      

      even better to

      from T where a = 1
      insert into T select ROW__ID SORT BY ROW__ID
      insert into T select a, 7
      

      but this won't parse currently.

      This is very similar to how MERGE stmt is handled.

      Need some though on on how WriteSet tracking works. If we don't allow updating partition column, then even with dynamic partitions TxnHandler.addDynamicPartitions() should see 1 entry (in Update type) for each partition since both the insert and delete land in the same partition. If part cols can be updated, then then we may insert a Delete event into P1 and corresponding Insert event into P2 so addDynamicPartitions() should see both parts. I guess both need to be recored in Write_Set but with different types. The delete as 'delete' and insert as insert so that it can conflict with some IOW on the 'new' partition.

      Attachments

        Activity

          People

            kkasa Krisztian Kasa
            ekoifman Eugene Koifman
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 19h
                19h