Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22098

Data loss occurs when multiple tables are join with different bucket_version

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Bug
    • Status: Patch Available
    • Blocker
    • Resolution: Unresolved
    • 3.1.0, 3.1.2
    • None
    • Operators


      When different bucketVersion of tables do join and no of reducers is greater than 2, the result is incorrect (data loss).
      Scenario 1: Three tables join. The temporary result data of table_a in the first table and table_b in the second table joins result is recorded as tmp_a_b, When it joins with the third table, the bucket_version=2 of the table created by default after hive-3.0.0, temporary data tmp_a_b initialized the bucketVerison=-1, and then ReduceSinkOperator Verketison=-1 is joined. In the init method, the hash algorithm of selecting join column is selected according to bucketVersion. If bucketVersion = 2 and is not an acid operation, it will acquired the new algorithm of hash. Otherwise, the old algorithm of hash is acquired. Because of the inconsistency of the algorithm of hash, the partition of data allocation caused are different. At stage of Reducer, Data with the same key can not be paired resulting in data loss.

      Scenario 2: create two test tables, create table table_bucketversion_1(col_1 string, col_2 string) TBLPROPERTIES ('bucketing_version'='1'); table_bucketversion_2(col_1 string, col_2 string) TBLPROPERTIES ('bucketing_version'='2');
      when use table_bucketversion_1 to join table_bucketversion_2, partial result data will be loss due to bucketVerison is different.



        1. table_c_data.orc
          0.4 kB
          GuangMing Lu
        2. table_b_data.orc
          0.4 kB
          GuangMing Lu
        3. table_a_data.orc
          0.4 kB
          GuangMing Lu
        4. join_test.sql
          0.6 kB
          GuangMing Lu
        5. image-2019-08-12-18-45-15-771.png
          8 kB
          GuangMing Lu
        6. HIVE-22098.1.patch
          3 kB
          GuangMing Lu

        Issue Links


          This comment will be Viewable by All Users Viewable by All Users


            Unassigned Unassigned Assign to me
            luguangming GuangMing Lu




                Issue deployment